Image processing apparatus and associated method

ABSTRACT

An image data processing apparatus is provided and includes a variable length decoding section to decode and extract quantized DCT coefficients from an encoded signal input thereto and extract encoded information from side information added to the encoded signal. Classification adaptation processing sections use the input signal supplied from a decoding section and a creation signal supplied from a signal storage section to determine the reliability of each of motion compensating vectors of the encoded signal supplied from a coded information storage section, selects that one of the motion compensating vectors which has the highest reliability as a selected vector, and produces a creation signal based on the selection vector.

CROSS REFERENCE

This application is a Division of and claims the benefit of priorityunder 35 USC §120 from U.S. Ser. No. 10/744,043, filed Dec. 24, 2003,and claims the benefit of priority under 35 USC §119 from JapanesePatent Application No. 2002-371403, filed Dec. 24, 2002, the entirecontents of each are incorporated herein by reference.

BACKGROUND OF THE INVENTION

The present invention relates to an image data processing apparatus andassociated method, a recording medium, and a program. More particularly,the present invention relates to an image data processing apparatus andmethod. A recording medium, and program by which the quality of an imagesignal can be enhanced.

A method of performing 2-3 pull-down of encoded images of a DVD (DigitalVersatile Disc) or the like such as 24P (Progressive) (24 frames per 60fields) images of a video signal or the like into 60i (Interlace) (60fields per second) or 60P (Progressive) (60 frames per second) images isknown as disclosed in Japan Laid-Open Patent Application No. HEI07-123291.

Additionally, in order to increase the number of frames, a temporalresolution creation process is sometimes used as disclosed in JapanLaid-Open Patent Application No. Hei 2002-199349.

Conventionally, in order to perform temporal resolution creation, motioncompensating vectors are determined from a creation (image) signalproduced once, and then the motion compensating vectors are utilized toperform a temporal resolution creation process.

However, since a creation signal produced contains coding distortion,the method of performing creation of a time resolution utilizing motioncompensating vectors determined from a creation signal cannot accuratelycalculate motion. Accordingly, there is a problem that presently knownmethods of temporal resolution creation do not provide an optimal imagequality.

SUMMARY OF THE INVENTION

It is an object of the present invention to improve the quality of animage signal.

According to a first aspect of the present invention, there is providedan image data processing apparatus, having a first selection means forselecting a motion compensating vector of a noticed region based onadditional information added to image data. A classification meansclassifies the noticed region into one of a plurality of classes basedon the motion compensating vector selected by the first selection means.A second selection means selects a prediction coefficient based on theclass classified by the classification means A construction meansconstructs a prediction tap of the noticed region. An arithmeticoperation means arithmetically operates a resolution creation signalbased on the selected prediction coefficient and the prediction tap.

The image data processing apparatus may be configured such that itfurther includes a first extraction means for extracting a plurality ofcandidate vectors of the motion compensating vector from the additionalinformation. A reliability calculation means calculates a reliability ofeach of the candidate vectors. An evaluation means evaluates thereliabilities of the candidate vectors calculated by the reliabilitycalculation means. The first selection means selects the candidatevectors evaluated to have the highest reliability by the evaluationmeans as the motion compensating vector of the noticed region.

The reliability calculation means may include re-encoding means forre-encoding an input signal of the noticed region. A second extractionmeans extractes re-encoded vectors corresponding to the candidatevectors from the re-encoded signal. A comparison means compares thecandidate vectors with the re-encoded vectors extracted by the secondextraction means to calculate the reliabilities of the candidatevectors.

The reliability calculation means may include neighborhood vectorextraction means for extracting neighborhood vectors corresponding tothe candidate vectors. A comparison means compares the candidate vectorswith the neighborhood vectors to calculate the reliabilities of thecandidate vectors.

The reliability calculation means may include history extraction meansfor extracting a vector history corresponding to each of the candidatevectors. A comparison means compares the candidate vectors with thecandidate vectors in the past obtained from the vector history tocalculate the reliabilities of the candidate vectors.

The reliability calculation means may include history extraction meansfor extracting a vector history corresponding to each of the candidatevectors. A discontinuity evaluation means calculates the reliability ofeach of the candidate vectors from a discontinuity in motion of thecandidate vector obtained from the vector history.

The reliability calculation means may include extraction means forextracting a motion compensating vector of an overall screen. Acomparison means compares the candidate vectors with the motioncompensating vector of the overall screen to calculate the reliabilitiesof the candidate vectors.

The reliability calculation means may calculate the reliabilities of thecandidate vectors based on motion compensation residuals of thecandidate vectors.

The first selection means may select, when the candidate vectorevaluated to have the highest reliability by the evaluation means cannotbe selected as the motion compensating vector of the noticed region,that one of the candidate vectors whose motion compensation residual isthe smallest as the motion compensating vector of the noticed region.

The image data processing apparatus may be configured such that itfurther includes tap extraction means for extracting the noticed regionas a tap based on the motion compensating vector selected by the firstselection means. The classification means classifies the noticed regioninto one of the plurality of classes based on a positional relationshipof the noticed region drawn near as a tap by the tap extraction meansand a boundary of the noticed region before drawn near as a tap by thetap extraction means.

An image data processing method of the first aspect includes selecting amotion compensating vector of a noticed region based on additionalinformation added to image data. A classification of the noticed regioninto one of a plurality of classes based on the motion compensatingvector selected by the process of the first selection step. A secondselection of a prediction coefficient based on the class classified bythe process of the classification step. A construction prediction tap ofthe noticed region, arithmetically operating a resolution creationsignal based on the selected prediction coefficient and the predictiontap.

A program of a recording medium of the first aspect includes a firstselection of a motion compensating vector of a noticed region based onadditional information added to image data. A classification of thenoticed region into one of a plurality of classes based on the motioncompensating vector selected by the process of the first selection step.A second selection of a prediction coefficient based on the classclassified by the process of the classification step. Constructing aprediction tap of the noticed region, and an arithmetic operation stepof arithmetically operating a resolution creation signal based on theselected prediction coefficient and the prediction tap.

A program of the first aspect causes a data processing device to selecta motion compensating vector of a noticed region based on additionalinformation added to image data. The data processor classifies thenoticed region into one of a plurality of classes based on the motioncompensating vector selected by the process of the first selection step.The data processor selects a prediction coefficient based on the classclassified by the process of the classification step. The data processorconstructs a prediction tap of the noticed region, and arithmeticallyoperates a resolution creation signal based on the selected predictioncoefficient and the prediction tap.

According to a further aspect, there is provided an image dataprocessing apparatus having a selection means for selecting a motioncompensating vector of a noticed region based on additional informationadded to student data. A classification means classifies the noticedregion into one of a plurality of classes based on the motioncompensating vector selected by the selection means. A learning meansconstructs a prediction tap of the noticed region based on the motioncompensating vector selected by the selection means and learning aprediction coefficient based on the class classified by theclassification means using teacher data corresponding to the constructedlearning tap.

The image data processing apparatus may be configured such that itfurther includes tap extraction means for extracting the noticed regionas a tap based on the motion compensating vector selected by theselection means. The classification means classifies the noticed regioninto one of the plurality of classes based on a positional relationshipof the noticed region drawn near as a tap by the tap extraction meansand a boundary of the noticed region before tap extraction as a tap bythe tap extraction means.

An image data processing method of the further aspect of the presentinvention includes selecting a motion compensating vector of a noticedregion based on additional information added to student data. The methodclassifies the noticed region into one of a plurality of classes basedon the motion compensating vector selected by the process of theselection. The method constructs a prediction tap of the noticed regionbased on the motion compensating vector selected by the process of theselection step and learning a prediction coefficient based on the classclassified by the process of the classification step using teacher datacorresponding to the constructed learning tap.

A program of a recording medium of the further aspect includes selectinga motion compensating vector of a noticed region based on additionalinformation added to student data, classifying the noticed region intoone of a plurality of classes based on the motion compensating vectorselected by the process of the selection step, and constructing aprediction tap of the noticed region based on the motion compensatingvector selected by the process of the selection step and learning aprediction coefficient based on the class classified by the process ofthe classification step using teacher data corresponding to theconstructed learning tap.

A program of the further aspect of the present invention causes acomputer to select a motion compensating vector of a noticed regionbased on additional information added to student data, classify thenoticed region into one of a plurality of classes based on the motioncompensating vector selected by the process of the selection step, andconstruct a prediction tap of the noticed region based on the motioncompensating vector selected by the process of the selection step andlearning a prediction coefficient based on the class classified by theprocess of the classification step using teacher data corresponding tothe constructed learning tap.

In the first aspect, a motion compensating vector of a noticed region isselected based on additional information added to image data, and thenoticed region is classified into one of a plurality of classes based onthe selected motion compensating vector. Then, a prediction coefficientis selected and a prediction tap of the noticed region is constructedbased on the class obtained by the classification. Next, a resolutioncreation signal is arithmetically operated based on the selectedprediction coefficient and the prediction tap.

In the further aspect of the present invention, a motion compensatingvector of a noticed region is selected based on additional informationadded to student data, and the noticed region is classified into one ofa plurality of classes based on the selected motion compensating vector.A prediction tap of the noticed region is constructed based on theselected motion compensating vector, and a prediction coefficient islearned based on the class obtained by the classification using teacherdata corresponding to the constructed prediction tap.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

FIG. 1 is a high level block diagram showing an exemplary configurationof an image data processing apparatus in accordance with the presentinvention;

FIG. 2 is a high level block diagram showing an exemplary configurationof a classification adaptation processing section of FIG. 1;

FIG. 3 is a block diagram showing an exemplary configuration of a shiftinformation extraction section of FIG. 2;

FIG. 4 is a block diagram showing an exemplary configuration of aclassification section of FIG. 2;

FIG. 5 is a block diagram showing an exemplary configuration of aprediction arithmetic operation section of FIG. 2;

FIGS. 6A and 6B are time charts illustrating an exemplary tap extractionwhere M=1;

FIG. 7 is a frame sequence of an exemplary creation of an intermediateframe where M=1;

FIG. 8 is a pixel grouping of an exemplary tap extraction where M=1;

FIG. 9 is a time chart showing another exemplary of tap extraction whereM=1;

FIG. 10 is a time chart showing a further exemplary tap extraction whereM=1;

FIG. 11 is a flow chart illustrating a creation process of theclassification adaptation processing section of FIG. 1;

FIG. 12 is a flow chart illustrating a shift amount calculation processat step S2 of FIG. 11;

FIGS. 13A and 13B are display orders illustrating an exemplary motioncompensating vector in the temporal direction where M=3;

FIGS. 14A and 14B are display orders of another exemplary motioncompensating vector in the temporal direction where M=3;

FIGS. 15A and 15B are motion vector charts illustrating an exemplarycreation of a time resolution;

FIGS. 16A and 16B are motion vector charts illustrating an exemplaryconventional creation of a time resolution;

FIGS. 17A and 17B are pixel maps illustrating an exemplary motioncompensating vector in the spatial direction;

FIG. 18 is a flow chart illustrating a reliability determination processat step S22 of FIG. 12;

FIG. 19 is a block diagram showing an example of configuration of acharacteristic amount extraction section of FIG. 3;

FIG. 20 is a flow chart illustrating a reliability calculation processat step S32 of FIG. 16;

FIG. 21 is a view showing an exemplary original image;

FIG. 22 is a view showing an exemplary image of a created intermediateframe;

FIG. 23 is a view illustrating an exemplary result of mapping ofdigitized reliabilities;

FIG. 24 is a block diagram showing another exemplary configuration ofthe characteristic amount extraction section of FIG. 3;

FIG. 25 is a flow diagram illustrating details of the reliabilitycalculation process executed by the characteristic amount detectionsection of FIG. 3;

FIG. 26 is a flow chart illustrating another exemplary reliabilitycalculation process at step S32 of FIG. 18;

FIG. 27 is a view illustrating an example of result of comparison ofmotion compensating vectors in a P picture;

FIG. 28 is a flow chart illustrating a motion compensating vectorselection process at step S23 of FIG. 12;

FIG. 29 is a time v. display order chart of a re-search process for amotion compensating vector;

FIG. 30 is a view illustrating candidate vectors the re-search processfor a motion compensating vector;

FIG. 31 is a block diagram showing another exemplary configuration ofthe shift information extraction section of FIG. 2;

FIG. 32 is a block diagram showing an exemplary configuration of acharacteristic amount extraction section of FIG. 31;

FIG. 33 is a flow chart illustrating another exemplary reliabilitycalculation process at step S32 of FIG. 18;

FIG. 34 is a view illustrating another exemplary mapping of digitizedreliabilities;

FIG. 35 is a view illustrating a further exemplary mapping of digitizedreliabilities;

FIG. 36 is a block diagram showing another exemplary configuration ofthe characteristic amount extraction section of FIG. 31;

FIG. 37 is a flow chart illustrating another exemplary reliabilitycalculation process at step S32 of FIG. 18;

FIG. 38 is a block diagram showing a further exemplary configuration ofthe characteristic amount extraction section of FIG. 31;

FIG. 39 is a flow chart illustrating a further exemplary reliabilitycalculation process at step S32 of FIG. 18;

FIG. 40 is a motion compensating vector history;

FIG. 41 is a block diagram showing another exemplary configuration ofthe characteristic amount extraction section of FIG. 3;

FIG. 42 is a flow chart illustrating another exemplary reliabilitycalculation process at step S32 of FIG. 18;

FIG. 43 is a block diagram showing a further exemplary configuration ofthe shift information extraction section of FIG. 2;

FIG. 44 is a flow chart illustrating another exemplary motioncompensating vector selection process at step S23 of FIG. 12;

FIG. 45 is a view showing another exemplary image of a createdintermediate frame;

FIG. 46 is a view illustrating another exemplary result of mapping ofdigitized reliabilities;

FIGS. 47A and 47B are pixel maps illustrating an exemplary tapextraction;

FIGS. 48A and 48B are exemplary configuration of class taps;

FIGS. 49A and 49B are patterns of a block boundary;

FIG. 50 is a view illustrating an exemplary configuration of a classcode;

FIG. 51 is a block diagram showing an exemplary configuration of alearning apparatus of the present invention;

FIG. 52 is a flow chart illustrating a learning process of the learningapparatus of FIG. 51;

FIG. 53 is a block diagram showing another exemplary configuration ofthe classification adaptation processing section of FIG. 1;

FIG. 54 is a block diagram showing an exemplary configuration of a shiftinformation extraction section of FIG. 53;

FIG. 55 is a flow chart illustrating an exemplary creation process ofthe classification adaptation processing section of FIG. 53;

FIG. 56 is a block diagram showing another exemplary configuration ofthe learning apparatus of FIG. 51;

FIG. 57 is a flow chart illustrating a learning process of the learningapparatus of FIG. 56;

FIG. 58 is a block diagram showing a further exemplary configuration ofthe classification adaptation processing section of FIG. 1;

FIG. 59 is a flow chart illustrating an exemplary creation process ofthe classification adaptation processing section of FIG. 58;

FIG. 60 is a block diagram showing a further exemplary configuration ofthe learning apparatus of FIG. 51;

FIG. 61 is a flow chart illustrating a learning process of the learningapparatus of FIG. 60; and

FIG. 62 is a block diagram showing another exemplary configuration ofthe image data processing apparatus of the present invention.

DETAILED DESCRIPTION OF THE INVENTION

In the following, an exemplary embodiment of the present invention isdescribed with reference to the drawings.

FIG. 1 shows a configuration of an image data processing apparatus 1 towhich the present invention is applied. It is to be noted that, whilethe image data processing apparatus 1 includes various features forother processes such as a process of encoding data, in the example shownin FIG. 1, the description of such features are omitted for convenienceof illustration and brevity.

The image data processing apparatus 1 includes a decoding section 11 anda creation section 12. In the image data processing apparatus 1, a codedsignal of the MPEG (Moving Picture Experts Group) 2 system of 30 P (30frames per second) images, for example, is decoded by the decodingsection 11, and a time resolution is created by tap extraction based onmotion compensation of a pixel of interest and 60 P (60 frames perminute) images are produced (creation) by the creation section 12.

The decoding section 11 includes a variable length decoding section 20,a dequantization section 21, an IDCT (Inverse Discrete Cosine Transform)section 22, a picture selection section 23, a frame memory 24, a motioncompensation prediction section 25, an adder 26 and a parameter controlsection 27. In the decoding section 11, the variable length decodingsection 20 quantizes a coded signal input thereto, decodes,demultiplexes and extracts a quantized DCT coefficient from thequantized coded signal and adds the quantized DCT coefficient to thecoded signal, and then decodes, demultiplexes and extracts quantizationcharacteristic information and coded information transmitted thereto asside information.

The dequantization section 21 receives the quantized DCT coefficient andthe quantization characteristic information extracted by the variablelength decoding section 20 as inputs thereto and dequantizes thequantized DCT coefficient based on the quantization characteristicinformation to restore the DCT coefficient. The IDCT (Inverse DiscreteCosine Transform) section 22 inverse discrete cosine transforms therestored DCT coefficient to calculate a pixel value. An output of theIDCT section 22 exhibits an actual pixel value where the image inquestion is an I (information) picture but exhibits a difference valuebetween corresponding pixel values where the image in question is a P(predicted) or B (bi-directional) picture.

The picture selection section 23 receives the coded informationdemultiplexed by the variable length decoding section 20 as an inputthereto and outputs an image signal from the adder 26 as a decodedsignal when the coded information input indicates an image (B picture)which is not referred to by any other picture (that is, which is notused for a motion compensation prediction process). However, when theinput coded information indicates an image (I picture or P picture)which is referred to by some other picture (that is, which is used for amotion compensation prediction process), the picture selection section23 outputs the image signal from the adder 26 as a decoded signal andbesides supplies the image signal to the frame memory 24 so that it maybe stored into the frame memory 24. The motion compensation predictionsection 25 performs motion compensation for the image data stored in theframe memory 24 based on a motion compensating vector. The adder 26 addsa signal from the motion compensation prediction section 25 to thedifference signal (signal of a P picture or a B picture) from the IDCTsection 22 and outputs a resulting signal to the picture selectionsection 23.

The parameter control section 27 receives a parameter A from an inputsection 416 (FIG. 62) based on an operation of a user. The parameter Ais, for example, a Volume value. The Volume value is a value for picturequality adjustment for making the horizontal resolution and the verticalresolution have suitable values at the same time, for making thehorizontal resolution, vertical resolution and noise removal degree havesuitable values at the same time, or the like. Details of the Volumevalue are disclosed, for example, in Japanese Patent Laid-Open No.2002-218414, the contents of which are incorporated by reference herein.Further, the parameter control section 27 provides a conversion functionusing a quantization scale of the quantization characteristicinformation and a bit rate of the coded information extracted by thevariable length decoding section 20 to determine a parameter B from theparameter A and supplies the parameter B to classification adaptiveprocessing sections 31-1 to 31-3.

For example, when “1.0” of the parameter A is arithmetically operatedwith a conversion function for use where the bit rate is “10 Mbps” andthe quantization scale is “40”, “1.00” of the parameter B is obtained.Meanwhile, when “1.0” of the parameter A is arithmetically operated witha conversion function for use where the bit rate is “10 Mbps” and thequantization scale is “20”, “0.50” of the parameter B is obtained. When“0.5” of the parameter A is arithmetically operated with a conversionfunction for use where the bit rate is “10 Mbps” and the quantizationscale is “40”, “0.50” of the parameter B is obtained. When “0.5” of theparameter A is arithmetically operated with a conversion function foruse where the bit rate is “10 Mbps” and the quantization scale is “20”,“0.25” of the parameter B is obtained.

In other words, the parameter B is obtained by conversion of theparameter A in accordance with the bit rate and the quantization scale.While the parameter A assumes a value within a range necessary forpicture quality adjustment as viewed from a user (human being), sincethe parameter B is given as a combination of the parameter A with thebit rate and the quantization scale, the number of values within therange is greater than that of the parameter A. As a result, fineradjustment can be achieved.

The creation section 12 includes classification adaptive processingsections 31-1 to 31-3, a coded information storage section 32 and asignal storage section 33. The classification adaptive processingsections 31-1 to 31-3 execute classification adaptation processes for anI picture, a P picture and a B picture, respectively, and fetch data ofan I picture, a P picture and a B picture from an input signal suppliedthereto from the picture selection section 23 based on coded informationfrom the coded information storage section 32. Since the classificationadaptive processing sections 31-1 to 31-3 have a basically sameconfiguration except that they process different pictures from oneanother, where there is no necessity to distinguish them from oneanother, each of them is referred to merely as classification adaptiveprocessing section 31.

Each classification adaptive processing section 31 determines thereliability of each of motion compensating vectors of coded informationextracted by the variable length decoding section 20 and stored in thecoded information storage section 32 using a decoded signal (hereinafterreferred to as input signal) input from the picture selection section 23of the decoding section 11 and a creation signal output from theclassification adaptive processing section 31 and stored once into thesignal storage section 33 and selects a predetermined motioncompensating vector based on the determined reliabilities. Where only amotion compensating vector having a high reliability is used in thismanner, a high quality temporal resolution process can be achieved. Theclassification adaptive processing section 31 arithmetically operates ashift amount of a noticed pixel (central tap) based on the selectedmotion compensating vector (selected vector), constructs a class tap bythe tap extraction from the input signal and the creation signal anddetermines a class in accordance with a characteristic amount thusdetermined. Further, the classification adaptive processing section 31constructs a prediction tap by the tap extraction from the input signaland the creation signal, executes a prediction arithmetic operationprocess for the prediction tap using prediction coefficient dataselected based on the class from within coefficient memories 71-0 to71-N (FIG. 2) corresponding to the parameter B from the parametercontrol section 27 to produce a creation signal of a time resolution andoutputs the creation signal.

The coded information storage section 32 stores coded information of aplurality of frames demultiplexed from the side information by thevariable length decoding section 20 of the decoding section 11. Thecoded information is formed from, for example, a motion compensatingvector and so forth. Consequently, upon classification, a motioncompensating vector can be referred to not only in a spatial direction(within the same frame) but also in a temporal direction (acrossdifferent frames), that is, in both of the spatial and temporaldirections.

The signal storage section 33 stores the input signal supplied theretofrom the picture selection section 23 and creation signals produced bythe classification adaptive processing sections 31. The input signal anda creation signal have a positional relationship between the present andthe past in time. Consequently, upon classification or upon predictiontap construction, signals in the temporal and spatial directions can bereferred.

FIG. 2 shows a first example of configuration of the classificationadaptive processing section 31 of FIG. 1.

A shift amount arithmetic operation section 51 includes a candidatevector acquisition section 61 and a shift information extraction section62. The candidate vector acquisition section 61 acquires motioncompensating vectors relating to a noticed pixel (motion compensatingpixels in the temporal direction and the spatial direction) from codedinformation stored in the coded information storage section 32.

The shift information extraction section 62 uses the input signalsupplied from the picture selection section 23 and the creation signalstored in the signal storage section 33 to evaluate the reliability ofcandidate vectors corresponding to the noticed pixel from among thecandidate vectors (motion compensating vectors) acquired by thecandidate vector acquisition section 61 to select that one of thecandidate vectors which is discriminated to have the highest reliabilityas a selected vector. Then, the shift information extraction section 62detects, based on the selected vector, shift information representativeof a shift amount of the noticed pixel (central tap), the reliability, amotion compensation residual or the like of the motion vector as a shiftamount detection characteristic amount and outputs the shift informationand the shift amount detection characteristic amount to a classificationsection 52 and a prediction arithmetic operation section 54.

The classification section 52 uses the input signal supplied from thepicture selection section 23 of the decoding section 11 and the creationsignal supplied from the signal storage section 33 to construct a classtap by the tap extraction based on the shift amount and the shift amountdetection characteristic amount supplied from the shift informationextraction section 62, detects a characteristic amount of the class tap,produces a class code based on the detected characteristic amount, andoutputs the class code to a prediction coefficient selection section 53.

The prediction coefficient selection section 53 selects, from among thecoefficient memories 71-0 to 71-N, that coefficient memory correspondingto the parameter B supplied from the parameter control section 27 of thedecoding section 11. Then, the prediction coefficient selection section53 selects, from among prediction coefficient data stored in advance inthe selected coefficient memory, that prediction coefficient data whichcorresponds to the class code produced by the classification section 52,and outputs the selected prediction coefficient data to the predictionarithmetic operation section 54. It is to be noted that such predictioncoefficient data are calculated by a learning apparatus 301 hereinafterdescribed with reference to FIG. 51 and stored in advance into thecoefficient memories 71-0 to 71-N in accordance with the parameter B.

The prediction arithmetic operation section 54 uses the input signalsupplied from the picture selection section 23 of the decoding section11 and the creation signal supplied from the signal storage section 33to construct a prediction tap by the tap extraction based on the shiftamount and the shift amount detection characteristic amount of thecentral tap supplied from the shift information extraction section 62,executes a prediction arithmetic operation process using the predictioncoefficient data from the prediction coefficient selection section 53based on the prediction tap to produce a creation signal of a createdtime resolution and outputs the creation signal.

FIG. 3 shows a first example of configuration of the shift informationextraction section 62 of FIG. 2.

In the example of FIG. 3, the shift information extraction section 62includes a reliability determination section 101 and a motioncompensating vector selection section 102. The reliability determinationsection 101 receives the candidate vectors acquired by the candidatevector acquisition section 61, the input signal supplied from thepicture selection section 23 of the decoding section 11 and the creationsignal supplied from the signal storage section 33 as input signalsthereto.

A characteristic amount extraction section (reliability arithmeticoperation section) 111 of the reliability determination section 101 usesthe input signal or the creation signal to calculate a motioncompensation residual of a candidate vector to determine the reliabilityof the candidate vector (for example, it is determined that the smallerthe motion compensation residual, the higher the reliability).Alternatively, the characteristic amount extraction section 111re-encodes the input signal or the creation signal, extracts a motioncompensating vector corresponding to a candidate vector from within there-encoded information, and compares such motion compensating vectors todetermine the reliabilities of the candidate vectors (for example, it isdetermined that the smaller the absolute value of the difference betweenthe two vectors, the higher the reliability). It is to be noted thatthis process is hereinafter described in detail with reference to FIG.26. A reliability evaluation section 112 discriminates whether or notthe reliabilities of the candidate vectors determined by thecharacteristic amount extraction section 111 are higher than apredetermined reference value, determines those candidate vectorsdetermined to have higher reliabilities than the reference value assignificant vectors and outputs the significant candidate vectors to themotion compensating vector selection section 102 together with thereliabilities of them.

The reliability determination section 101 includes a re-search section121, a characteristic amount extraction section 122 and a motioncompensating vector decision section 123. The re-search section 121 usesthe input signal or the creation signal to take elements around a startpoint, around an end point or around a midpoint of each of thesignificant candidate vectors output from the reliability evaluationsection 112 into consideration to determine a weighted mean to re-searchwhether or not there is a motion compensating vector which is moresuitable (has a higher reliability). If a more suitable motioncompensating vector is found, then the re-search section 121 outputs themotion compensating vector to the characteristic amount extractionsection 122 (processing of FIGS. 8 and 29 hereinafter described).

The characteristic amount extraction section (reliability arithmeticoperation section) 122 uses the input signal or the creation signal tocalculate motion compensation residuals of the candidate vectors basedon the re-searched significant candidate vectors and the reliabilitiesof them to determine the reliabilities of the candidate vectors. Themotion compensating vector decision section 123 selects, from among thesignificant candidate motion vectors, that candidate vector which hasthe highest reliability as a selected vector and determines a shiftamount of the noticed pixel (central tap) based on the selected vector.Further, the motion compensating vector decision section 123 outputs ashift amount detection characteristic amount such as, for example, thereliability or the motion compensation residual of the determinedcandidate vector to the classification section 52 and the predictionarithmetic operation section 54 together with the shift amount of thecentral tap.

FIG. 4 shows an example of a configuration of the classification section52 of FIG. 2.

The classification section 52 includes a class tap construction section131 and a class code production section 132. The class tap constructionsection 131 determines a creation pixel by the tap extraction from theinput signal and the creation signal based on the shift amount and theshift amount detection characteristic amount from the motioncompensating vector decision section 123, constructs a class tap (pixel)corresponding to the creation pixel necessary for execution ofclassification and outputs the class tap to the class code productionsection 132.

The class code production section 132 extracts a characteristic amountof the class tap constructed by the class tap construction section 131based on the shift amount and the shift amount detection characteristicamount. The characteristic amount may be, for example, a shift amount(direction, magnitude) of the central tap, a motion compensationresidual from within the shift amount detection characteristic amount, areliability of a motion compensating vector from within the shift amountdetection characteristic amount, a pattern of a block boundary upon thetap extraction or the like. Further, the class code production section132 decides a class of the creation pixel based on a threshold value setin advance or the like in accordance with the extracted characteristicamount of the class tap, produces a class code of the decided class andoutputs the class code to the prediction coefficient selection section53.

FIG. 5 shows an example of a configuration of the prediction arithmeticoperation section 54 of FIG. 2.

The prediction arithmetic operation section 54 includes a prediction tapconstruction section 141 and an arithmetic operation section 142. Theprediction tap construction section 141 determines a creation pixel bythe tap extraction from the input signal and the creation signal basedon the shift amount and the shift amount detection characteristic amountfrom the motion compensating vector decision section 123, constructs aprediction tap (pixel) corresponding to the creation pixel and outputsthe prediction tap to the arithmetic operation section 142.

The arithmetic operation section 142 multiplies the prediction tap fromthe prediction tap construction section 141 by the predictioncoefficient data determined by the prediction coefficient selectionsection 53 based on the shift amount and the shift amount detectioncharacteristic amount from the motion compensating vector decisionsection 123 to execute a prediction arithmetic operation process toproduce a creation signal of a time solution and outputs the creationsignal.

Now, a principle of a creation process of a time resolution carried outin the present invention is described with reference to FIGS. 6 to 10.It is to be noted that, in the example of FIGS. 6 to 10, images of 30 Pare converted into images of 60 P through creation of a time resolutionand M is M=1 where the GOP structure includes no B picture (a structureof I, P, P, P, . . . pictures).

FIG. 6 illustrates frames at times t and t−1. In the example of FIG. 6,a partition between frames indicates a macro block. First, as shown inFIG. 6A, a macro block m1 to which a noticed pixel al at time t belongsis determined, and a macro block type, a prediction method and a motioncompensating vector MV 1-1 of the macro block m1 of the noticed pixel a1are acquired from the coded information of the side information. Then, areference pixel b1 at time t−1 is determined based on the acquiredprediction method.

Then, as seen in FIG. 6B, a creation pixel c1 at time t−½ intermediatebetween time t1, at which motion compensation (MC) is performed with amotion compensating vector MV1-2, and time t−1 is determined from thereference pixel b1 of at the determined time t−1 (in simpledetermination, MV1-2=(½)MV1-1 is used). The creation pixel cl at timet−½ (t−1<t−½<t) is determined in such a manner as described above, andan intermediate frame from the creation pixel c1 is created. In otherwords, creation of a time resolution by the tap extraction (detailedmeaning of which is hereinafter described with reference to FIG. 47).

Further, where M=1, since the I picture and the P pictures among thepictures appearing in order as I, P, P, . . . , P are displayed in thisorder, in order to convert 30 P images into 60 P images through creationof a time resolution as seen in FIG. 7, motion compensation is performedbased on the motion compensating vectors each beginning with the Ipicture and ending with a P picture, and an intermediate frame f1,another intermediate frame f2, . . . are created between the frames ofthe pictures and displayed.

In the example of FIG. 7, frames appearing in order in a bit stream areshown in the upper stage, and frames appearing in a displaying order areshown in the lower stage. An arrow mark below each of the picturesappearing in the displaying order in the lower stage indicates acandidate vector in the temporal direction which can be used as a motioncompensating vector. The start point of each arrow mark indicates aframe at the start point of a motion vector in the temporal direction (aframe including the reference pixel) while the end point of the arrowmark indicates a frame at the end point of the candidate vector in thetemporal direction (a frame including the noticed pixel). Further, anexemplary numerical value (½) appearing on each arrow mark indicates avalue for internally dividing the motion compensating vector forcreation of an intermediate frame. In particular, in order to create anew frame intermediately between a frame and a next frame, a motioncompensating vector obtained by multiplying the value of the motioncompensating vector between the two frames by ½ is used. It is to benoted that the numerical value for the internal division is changed inaccordance with the creation position of the intermediate frame, andthose skilled in the art will recognize that alternative values may beutilized.

Further, in order to execute the creation of a time resolution with ahigher degree of accuracy, a process described below is executed inaddition to the process described above. For example, after a creationpixel c1 (FIG. 6B) is determined as shown in FIG. 8, block matching isperformed again, for example, for a pixel block BE1 of 5×5 pixels in theproximity of the reference pixel b1, in a unit of a pixel based on, forexample, a block AE1 of 3×3 pixels around the noticed pixel a1, and anappropriate reference pixel is determined from among the pixels. Then,the creation pixel c1 is re-produced again from the reference pixel andthe noticed pixel (process of the re-search section 121 of FIG. 3).

Meanwhile, in an example of FIG. 9, similarly as in determination of thecreation pixel c1 of FIG. 6, a macro block m2 to which a noticed pixela2 at time t belongs (a macro block displaced leftwardly by a two-macroblock distance from the macro block ml) is determined, and a motioncompensating vector MV2-1 is acquired from the coded information of theside information, whereafter a reference pixel b2 at time t−1 isdetermined. Then, a creation pixel c2 motion-compensated with the motioncompensating vector MV2-1 is determined from the reference pixel b2 attime t−1 thus determined (time t−½). In this instance, a range “e”within which motion compensation is difficult may possibly appear at aportion of time t−½ corresponding to a portion between the macro blockm1 and the macro block m2 (boundary between the blocks).

In such an instance, when the creation pixel c1 is to be created, theblock BE1 in the proximity of the reference pixel b1 is utilized tocreate a pixel (for example, a pixel c3) of a block CE1 including pixelsin the proximity of the creation pixel c1. If such a process as justdescribed is performed also for the macro block m2, then a pixel can becreated within the range e within which motion compensation isdifficult.

A creation process of a time resolution is executed using a motioncompensating vector in such a manner as described above. However, if aclassification adaptive process is not performed, then since a motioncompensating vector is calculated unnaturally on the boundary between amoving part and a still part, a bad influence may be had on the creationof an intermediate frame. Further, since an intra-block in a predictionframe (B or P picture) does not involve a motion compensating vector, atime resolution cannot be created for the intra-block. Furthermore,since a motion compensating vector is present only in a unit of a macroblock composed of 16×16 pixels, if a process for creation of a timeresolution results in failure, then replacement with another blockhaving no relation occurs, resulting in extreme deterioration of thepicture quality.

Therefore, in the present invention, a classification adaptive processis used to perform a creation process.

Now, operation is described. The variable length decoding section 20 ofthe decoding section 11 decodes and demultiplexes a quantized DCTcoefficient from within a coded signal transmitted thereto from a codingapparatus not shown and demultiplexes quantization characteristicinformation from within side information transmitted together with thequantized DCT coefficient, and outputs the quantized DCT coefficient andthe quantization characteristic information to the dequantizationsection 21. The dequantization section 21 dequantizes the quantized DCTcoefficient based on the quantization characteristic information. TheIDCT section 22 IDCT processes the DCT coefficient dequantized by thedequantization section 21 to decode the original signal.

The motion compensation prediction section 25 uses a motion compensatingvector included in the coded information decoded and extracted by thevariable length decoding section 20 to perform motion compensation foran image stored in the frame memory 24 and outputs a resulting image tothe adder 26. The adder 26 adds the signal motion-compensated by themotion compensation prediction section 25 to the signal output from theIDCT section 22 to produce locally decoded data and supplies the locallydecoded data to the picture selection section 23. The picture selectionsection 23 selects the data from the adder 26 based on the codedinformation and outputs the selected data to an apparatus in asucceeding stage not shown and supplies necessary data from within thelocally decoded data to the frame memory 24 so that the data may bestored into the frame memory 24.

The parameter control section 27 arithmetically operates a functionbased on the quantization characteristic information and the codedinformation decoded and extracted from the side information by thevariable length decoding section 20 and converts a parameter A input bya user into a parameter B based on the function.

The signal storage section 33 cumulatively stores an input signal outputfrom the picture selection section 23. Further, the coded informationstorage section 32 cumulatively stores the coded information decoded andextracted from the side information by the variable length decodingsection 20.

Each of the classification adaptive processing sections 31-1, 31-2 and31-3 performs a classification adaptive process to produce a creationsignal for a frame of an I picture, a P picture or a B picture,respectively, and outputs the produced creation signal.

The creation process of the exemplary embodiment is described below withreference to a flow chart of FIG. 11. It is to be noted that thecreation process is described briefly with reference to FIG. 11 so thata flow of the creation process may be recognized, and details ofprocesses at different steps are hereinafter described successivelyreferring to FIG. 12 and so forth.

The shift amount arithmetic operation section 51 of the classificationadaptive processing section 31 receives, as inputs thereto, the inputsignal supplied from the picture selection section 23 of the decodingsection 11 and the creation signal supplied from the signal storagesection 33. The input signal and the creation signal have a positionalrelationship between the present and the past in time. It is to be notedthat the creation signal may include a plurality of frames.

The shift amount arithmetic operation section 51 waits at step S1 untilthe input signal is input from the picture selection section 23 of thedecoding section 11. If the input signal and the creation signal areinput, then the shift amount arithmetic operation section 51 executes ashift amount calculation process (details of which are hereinafterdescribed with reference to a flow chart of FIG. 12) at step S2.

At step S3, the class tap construction section 131 of the classificationsection 52 receives, as inputs thereto, the input signal supplied fromthe decoding section 11 and the creation signal supplied from the signalstorage section 33, determines a creation pixel by the tap extractionbased on the shift amount and the shift amount detection characteristicamount supplied from the motion compensating vector decision section123, and constructs a class tap in accordance with the creation pixel.

At step S4, the class code production section 132 extracts acharacteristic amount of the class tap constructed by the class tapconstruction section 131 based on the shift amount and the shift amountdetection characteristic amount from the motion compensating vectordecision section 123.

At step S5, the class code production section 132 determines the classof the tap (pixel) based on a threshold value set in advance or the likein accordance with the extracted characteristic amount of the class tap,produces a class code and outputs the class code to the predictioncoefficient selection section 53.

At step S6, the prediction coefficient selection section 53 selects thatone of the coefficient memories 71-0 to 71-N which corresponds to theparameter B supplied from the parameter control section 27 of thedecoding section 11, selects that one of prediction coefficient datastored in advance in the selected coefficient memory which correspondsto the class code produced by the classification section 52, and outputsthe selected prediction coefficient data to the prediction arithmeticoperation section 54.

At step S7, the prediction tap construction section 141 of theprediction arithmetic operation section 54 receives, as inputs thereto,the input signal from the decoding section 11 and the creation signalstored in the signal storage section 33, determines a creation pixel bythe tap extraction based on the shift amount and the shift amountdetection characteristic amount supplied from the motion compensatingvector decision section 123, constructs a prediction tap in accordancewith the creation pixel and outputs the prediction tap to the arithmeticoperation section 142.

At step S8, the arithmetic operation section 142 performs a predictionarithmetic operation process for the prediction tap constructed at stepS7 using the prediction coefficient data selected by the predictioncoefficient selection section 53 to produce a creation signal of acreated time resolution, and outputs the creation signal.

The creation process of the present invention is executed in such amanner as described above to produce a creation signal of a created timeresolution.

In the following, the processes at the steps described above aresuccessively described more particularly. It is to be noted that theprocess at step S2 described above is described in detail with referenceto FIGS. 12 to 46, and the processes at steps S3 to S8 are described indetail with reference to FIGS. 47 to 50.

First, the shift amount calculation process at step S2 of FIG. 11 isdescribed with reference to a flow chart of FIG. 12.

At step S21, the candidate vector acquisition section 61 of the shiftamount arithmetic operation section 51 acquires motion compensatingvectors in the temporal direction and the spatial direction as candidatevectors from within the coded information supplied from the codedinformation storage section 32.

Motion compensating vectors in the temporal direction where M=3 (in thecase of IBBPBB . . . pictures) are described with reference to FIGS. 13and 14. FIG. 13A illustrates an array of frames in the temporaldirection in an ordinary case, and FIG. 13B illustrates a relationshipbetween the array of frames in the temporal direction in the ordinarycase and motion compensating vectors when intermediate frames arecreated. Further, FIG. 14A illustrates an array of frames in thetemporal direction in an exceptional case, and FIG. 14B illustrates arelationship between the array of frames in the temporal direction andmotion compensating vectors in the exceptional case when intermediateframes are created. Numerals subordinate to the characters I, P and Bindicate order numbers upon display of the images.

It is to be noted that, similarly as in the example of FIG. 7, in FIGS.13B and 14B, an arrow mark directed horizontally below each of thepictures arranged in the displaying order in the lower stage indicates amotion compensating vector in the temporal direction which can be usedfor creation of a time resolution (that is, which can become a candidatevector). Further, the start point of each arrow mark indicates the startpoint of a motion compensating vector in the temporal direction whilethe end point of the arrow mark indicates the end point of the motioncompensating vector in the temporal direction. Further, an exemplarynumerical value denoted on each arrow mark indicates a value forinternally dividing the motion compensating vector for creation of anintermediate frame (that is, a value for weighting). The numerical valueis changed in accordance with the creation position of the intermediateframe.

In the example of FIG. 13A, after two B pictures (B4 and B5 pictures)following a first I picture (I3 picture) within the bit streamillustrated in the upper stage are displayed, a next P picture (P6picture) is displayed. Further, after further two B pictures (B7 and B8)pictures are displayed, another I picture (I9 picture) id displayed.Different from the case wherein M=1 described above with reference toFIG. 7, the frame order (coding order) (I3, B1, B2, P6, B4, B5, I9, B7,B8, . . . ) in the bit stream in the upper stage and the frame order(B1, B2, I3, B4, B5, B6, B7, B8, I9, . . . ) in the lower stage when theframes are displayed actually are different from each other.

An intermediate frame is created in such a manner as describedhereinabove with reference to FIG. 6B between each adjacent ones offrames of such pictures as described above. In the example of FIG. 13B,in order to create an intermediate frame f11 between a frame of the I3picture and a frame of the B4 picture, at least one of a value equal toone half a motion compensating vector in the temporal direction whosestart point is the I3 picture and whose end point is the B4 picture,another value equal to ¼ a motion compensating vector in the temporaldirection whose start point is the I3 picture and whose end point is theB5 picture and a further value equal to ⅙ a motion compensating vectorin the temporal direction whose start point is the I3 picture and whoseend point is the P6 picture can be used.

Similarly, in order to create an intermediate frame f12 between a frameof the B4 picture and a frame of the B5 picture, at least one of a valueequal to ¾ a motion compensating vector in the temporal direction whosestart point is the P6 picture and whose end point is the B4 picture,another value equal to ¾ a motion compensating vector in the temporaldirection whose start point is the I3 picture and whose end point is theB5 picture and a further value equal to 1/2 a motion compensating vectorin the temporal direction whose start point is the I3 picture and whoseend point is the P6 picture can be used. Further, in order to create anintermediate frame f13 between a frame of the B5 picture and a frame ofthe P6 picture, at least one of a value equal to ¼ a motion compensatingvector in the temporal direction whose start point is the P6 picture andwhose end point is the B4 picture, another value equal to ½ a motioncompensating vector in the temporal direction whose start point is theP6 picture and whose end point is the B5 picture and a further valueequal to ⅚ a motion compensating vector in the temporal direction whosestart point is the I3 picture and whose end point is the P6 picture canbe used.

Furthermore, in order to create an intermediate frame f14 between aframe of the P6 picture and a frame of the B7 picture, at least one of avalue equal to ½ a motion compensating vector in the temporal directionwhose start point is the P6 picture and whose end point is the B7picture and another value equal to ¼ a motion compensating vector in thetemporal direction whose start point is the P6 picture and whose endpoint is the B8 picture. In order to create an intermediate frame f15between a frame of the B7 picture and a frame of the B8 picture, atleast one of a value equal to ¾ a motion compensating vector in thetemporal direction whose start point is the I9 picture and whose endpoint is the B7 picture and another value equal to ¾ a motioncompensating vector in the temporal direction whose start point is theP6 picture and whose end point is the B8 picture. Meanwhile, in order tocreate an intermediate frame f16 between a frame of the B8 picture and aframe of the I9 picture, at least one of a value equal to ¼ a motioncompensating vector in the temporal direction whose start point is theI9 picture and whose end point is the B7 picture and another value equalto ½ a motion compensating vector in the temporal direction whose startpoint is the I9 picture and whose end point is the B8 picture.

On the other hand, in the example of FIG. 14A, an I picture (I1 picture)is displayed first, and then a B picture (B2 picture) is displayed,whereafter a P picture (P3 picture) is displayed. Further, two Bpictures (B4 and B5 pictures) are displayed, and then a next P6 pictureis displayed. Similarly as in the example of FIG. 13A, the frame order(coding order) (I1, P3, B2, P6, B4, B5, . . . ) in the bit stream in theupper stage and the frame order (I1, B2, P3, B4, B5, P6, . . . ) in thelower stage when the frames are displayed actually are different fromeach other.

An intermediate frame is created in a similar manner as described abovewith reference to FIG. 13B between each adjacent ones of such frames ofthe pictures as just described. In the example of FIG. 14B, in order tocreate an intermediate frame f21 between a frame of the I1 picture and aframe of the B2 picture, at least one of a value equal to ½ a motioncompensating vector in the temporal direction whose start point is theI1 picture and whose end point is the B2 picture and another value equalto ¼ a motion compensating vector in the temporal direction whose startpoint is the I1 picture and whose end point is the P3 picture. In orderto create an intermediate frame f22 between a frame of the B2 pictureand a frame of the P3 picture, at least one of a value equal to ¾ amotion compensating vector in the temporal direction whose start pointis the P3 picture and whose end point is the B2 picture and anothervalue equal to ¾ a motion compensating vector in the temporal directionwhose start point is the I1 picture and whose end point is the P3picture.

As described above, where M=3, in order to create an intermediate framewith reference to a B picture, a plurality of, three in the maximum,motion compensating vectors existing in the temporal direction can beused for creation of a time resolution.

While the conversion of 30 P images into 60 P images through creation ofa time resolution is described above, conversion of 24 P images into 60P images through creation of a time resolution using a value forinternally dividing a motion compensating vector between frames at thestart point and the end point (that is, a value for weighting) isdescribed.

FIG. 15 illustrates an array of frames in the temporal direction. In theexample of FIG. 15, a frame of a solid line indicates an original frameof the 24 P images while a frame of a broken line indicates a frame ofthe 60 P images. Each arrow mark extending between different framesindicates one of motion compensating vectors MV0 to MV4 in the temporaldirection which can be used for creation of a time resolution (that is,which can become candidate vectors). The start point of each arrow markindicates the start point of a motion compensating vector in thetemporal direction while the end point of the arrow mark indicates theend point of the motion compensating vector in the temporal direction.Further, an exemplary numerical value shown above or below each frameindicates a value for internally dividing a motion compensating vectorfor creation of a frame between the frames at the start point and theend point (that is, a value for weighting). This value varies inaccordance with the creation position of the frame to be created.

In the example of FIG. 15A, creation of a temporal resolution isperformed based on six original frames of the 24 P images using valuesfor internal division of motion compensating vectors to create 15 framesof the 60 P images. In the example of FIG. 15A, between the start pointand the end portion of the motion compensating vector MV0 between thefirst original frame and the second original frame, a frame is createdby internally dividing (weighting) the motion compensating vector MV0 by(with) 5/14, and a next frame is created by internally dividing themotion compensating vector MV0 by 5/7. Between the start point and theend point of the motion compensating vector MV1 between the secondoriginal frame and the third original frame, the motion compensatingvector MV1 is internally divided by 1/14 to create a frame andinternally divided by 3/7 to create a next frame, and is furtherinternally divided by 11/14 to create another next frame.

Similarly, between the start point and the end portion of the motioncompensating vector MV2 between the third original frame and the fourthoriginal frame, the motion compensating vector Mv2 is internally dividedby 1/7 to create a frame and internally divided by ½ to create a nextframe, and is further internally divided by 6/7 to create another nextframe. Between the start point and the end portion of the motioncompensating vector MV3 between the fourth original frame and the fifthoriginal frame, the motion compensating vector MV3 is internally dividedby 3/14 to create a frame and internally divided by 4/7 to create a nextframe, and is further internally divided by 13/14 to create another nextframe. Between the start point and the end portion of the motioncompensating vector MV4 between the fifth original frame and the sixthoriginal frame, the motion compensating vector MV4 is internally dividedby 2/7 to create a frame and internally divided by 9/14 to create a nextframe.

In this manner, using the six original frames of the 24 P images, a timeresolution is created based on the individual motion compensatingvectors and 15 frames of the 60 P images are created.

On the other hand, where creation of a time resolution is performedbased on the motion compensating vector whose start point and end pointare the first original frame and the sixth original frame, respectively,as seen in FIG. 15B, frames between them are created through weightingwith 1/14, 2/14, 3/14, . . . , 13/14. Naturally, use of shorter motioncompensating vectors as seen in FIG. 15A realizes smoother motion havinga higher degree of continuity.

A DVD (Digital Versatile Disk) or the like has 24 P images recordedthereon, and upon reproduction, a DVD player converts the 24 P imagesinto and reproduces 60 P images. In this instance, images which exhibita smooth variation can be reproduced by converting 24 P images into 60 Pimages through creation of a temporal resolution using a value withwhich a motion compensating vector is internally divided.

FIG. 16 illustrates continuities in motion in a case wherein motioncompensating vectors are weighted to perform a time resolution processin such a manner as described above (FIG. 16A) and another case whereincreation of a time resolution is performed by a 2-3 pull-down process(FIG. 16B).

FIG. 16A illustrates the continuity in the case wherein a motioncompensating vector between each adjacent ones of original frames isweighted to perform creation of a time resolution as illustrated in FIG.15A. The images exhibit a smooth and continuous variation.

In contrast, since the 2-3 pull-down process creates three originalframes from one original frame as seen in FIG. 16B, no motion isexhibited among the three frames. Since two original frames are createdfrom a next one original frame, no motion is exhibited between the twooriginal frames. Some motion is exhibited only between a unit of threeframes and a unit of two frames. Accordingly, the continuity in motionis lost as seen from FIG. 16B. Even if it is tried to create a timeresolution from 60 P images created in this manner, it is difficult torecover continuous motion.

From the foregoing, according to the present invention, a value forinternally dividing a motion compensating vector is used to create atime resolution to convert 24 P images into 60 P images, and therefore,smoother motion having a higher degree of continuity is realized.

Subsequently, a motion compensating vector in a spatial direction isdescribed with reference to FIG. 17. FIG. 17 shows macro blocks Q0 to Q8each formed from 16×16 pixels on a frame having an axis of abscissaindicated by a horizontal direction and an axis of ordinate indicated bya vertical direction.

In the example of FIG. 17A, the macro block Q0 corresponds to the motioncompensating vector MV0 in the spatial direction; the macro block Q1corresponds to the motion compensating vector MV1 in the spatialdirection; the macro block Q2 corresponds to the motion compensatingvector MV2 in the spatial direction; the macro block Q3 corresponds tothe motion compensating vector MV3 in the spatial direction; the macroblock Q5 corresponds to the motion compensating vector MV5 in thespatial direction; the macro block Q6 corresponds to the motioncompensating vector MV6 in the spatial direction; the macro block Q7corresponds to the motion compensating vector MV7 in the spatialdirection; and the macro block Q8 corresponds to the motion compensatingvector MV8 in the spatial direction.

However, since the macro block Q4 is an intra block, it corresponds tono motion compensating vector in the spatial direction.

Therefore, as shown in FIG. 17B, the motion compensating vectors in thespatial direction obtained from the macro block Q0 to Q3 and Q5 to Q8around the macro block Q4 are utilized for the macro block Q4. Forexample, for a block of 8×8 pixels at a left upper portion of the macroblock Q4, the motion compensating vector MV0 of the macro block Q0leftwardly above the portion of the macro block Q4, the motioncompensating vector MV1 of the macro block Q1 above the portion of themacro block Q4 or the motion compensating vector MV3 of the macro blockQ3 leftwardly of the portion of the macro block Q4 is used instead.Similarly, for a block of 8×8 pixels at a right upper portion of themacro block Q4, the motion compensating vector MV1 of the macro block Q1above the portion of the macro block Q4, the motion compensating vectorMV2 of the macro block Q2 rightwardly above the portion of the macroblock Q4 or the motion compensating vector MV5 of the macro block Q5rightwardly of the macro block Q4 is used instead. For a block of 8×8pixels at a left lower portion of the macro block Q4, the motioncompensating vector MV3 of the macro block Q3 leftwardly of the portionof the macro block Q4, the motion compensating vector MV6 of the macroblock Q6 leftwardly below the portion of the macro block Q4 or themotion compensating vector MV7 of the macro block Q7 below the portionof the macro block Q4 is used instead. For a block of 8×8 pixels at aright lower portion of the macro block Q4, the motion compensatingvector MV5 of the macro block Q5 rightwardly of the portion of the macroblock Q4, the motion compensating vector MV7 of the macro block Q7 belowthe portion of the macro block Q4 or the motion compensating vector MV8of the macro block Q8 rightwardly below the macro block Q4 is usedinstead. This improves the picture quality on the boundary of a movingobject or around an intra block.

Further, while the above description is given of the macro block Q4 ofan intra block, the foregoing can be applied also to the other macroblocks. Accordingly, through utilization of motion compensating vectorsof surrounding macro blocks in the spatial direction, the accuracy of amotion compensating vector can be improved from a unit of 16×16 pixelsto another unit of 8×8 pixels, and the picture quality can be improvedas much.

A predetermined plural number of motion compensating vectors in thetemporal direction and the spatial direction are acquired in such amanner as described above.

Referring back to FIG. 12, after the candidate vector acquisitionsection 61 outputs the plurality of motion compensating vectors in thetemporal direction and the spatial direction acquired in such a manneras described above as candidate vectors to the shift informationextraction section 62, the reliability determination section 101 of theshift information extraction section 62 executes a reliabilitydetermination process at step S22 to determine, from among the candidatevectors in the temporal direction and the spatial direction, thosecandidate vectors which are to be utilized for creation of a timeresolution (tap extraction). The reliability determination process isdescribed with reference to a flow chart of FIG. 18.

At step S31, the characteristic amount extraction section 111 of thereliability determination section 101 discriminates whether or not acandidate vector to be processed is present. If it is discriminated thata candidate vector to be processed is present, then the characteristicamount extraction section 111 executes a reliability calculation processat step S32. Here, the characteristic amount extraction section 111includes a motion compensation residual calculation section 161 and amotion compensation residual evaluation section 162 as seen in FIG. 19.

Now, the reliability calculation process is described with reference toa flow chart of FIG. 20.

At step S41, the motion compensation residual calculation section 161uses the input signal from the picture selection section 23 of thedecoding section 11 and the creation signal from the signal storagesection 33 supplied at step S1 of FIG. 11 to calculate a motioncompensation residual from a candidate motion vector supplied from thecandidate vector acquisition section 61. At step S42, the motioncompensation residual evaluation section 162 compares and evaluates themotion compensation residual calculated by the motion compensationresidual calculation section 161 with a predetermined reference value oranother motion compensation residual from another candidate vector todetermine the reliability of the candidate vector and outputs thedetermined reliability to the reliability evaluation section 112.Thereafter, the processing advances to step S33 of FIG. 18. By thecomparison and evaluation, a high evaluation is provided, for example,to a candidate vector which exhibits the lowest motion compensationresidual value.

FIG. 21 illustrates an example of an original image where a person at acentral location moves rightward and a camera moves (pans) rightwardfollowing the person. In the example of FIG. 21, a wall having round andtriangular patterns thereon is present on the background. Since thecamera is panning, also the background moves rightward similarly to theperson. If the original image is used and motion compensation residualsfrom candidate vectors are calculated at step S41 of FIG. 20 and then acandidate vector whose motion compensation residual has the lowest valueamong the motion compensation residuals is determined as a selectedvector and creation of a time resolution is executed based on theselected vector, then such an image of an intermediate frame as shown inFIG. 22 is created. It is to be noted that, in FIG. 22, M=3.

In the example of FIG. 22, edge portions of the background of the roundsand triangles and boundary portions between the background and theperson exhibit considerable failure when compared with the example ofFIG. 21. This arises from the fact that, where M=3, the distance betweenframes is considerably long when compared with that where M=1. Throughcomparison of the image of the created intermediate frame and therelationship of the utilized motion compensating vectors, it can berecognized that a motion compensating vector which is fluctuating (thatmotion compensating vector which is significantly different in magnitudeor direction from the other neighboring motion compensating vectors) hasa bad influence on an edge portion of the background of the rounds andtriangles and the boundary portions between the background and theperson.

Thus, if only those significant motion compensating vectors determinedto have high reliabilities through execution of determination of eachmotion compensating vector in advance at step S33 of FIG. 18 are usedfor creation of a time resolution without using those motioncompensating vectors which have a bad influence, then a good image of anintermediate frame can be produced.

In particular, at step S33, the reliability evaluation section 112discriminates whether or not the reliability of the candidate vectordetermined at step S42 of FIG. 20 is higher than a reference value setin advance. If it is discriminated that the reliability of the candidatevector is higher than the reference value set in advance, thereliability evaluation section 112 outputs the candidate vector as asignificant vector to the motion compensating vector selection section102 together with the reliability of the candidate vector at step S34.Thereafter, the processing returns to step S31 to repeat the processesat the steps beginning with step S31.

If it is discriminated at step S33 that the reliability of the candidatevector is lower than the reference value set in advance, then theprocess at step S34 is not executed (but is skipped), and the processingreturns to step S31 so that the processes at the steps beginning withstep S31 are repeated. In other words, any candidate vector whosereliability is lower than the reference value set in advance is excludedas a motion compensating vector having a bad influence.

If it is discriminated at step S31 that a candidate vector to beprocessed is not present, that is, if it is discriminated that theprocess for all of the candidate vectors is completed, then theprocessing advances to step S23 of the flow chart of FIG. 12.

FIG. 23 illustrates a result when, where the original image of FIG. 21is used and a candidate vector whose motion vector residual exhibits thelowest value is determined as a selected vector to produce the image ofthe intermediate frame of FIG. 22, the reliability of the selectedvector is converted into a numeric value and mapped to the originalimage of FIG. 21.

In the example of FIG. 23, it is shown that a portion (block) indicatedby a thicker line has a lower reliability on the motion compensatingvector corresponding to the block. It can be recognized throughcomparison between FIGS. 21 and 23 that a thick line is present alongthe contour of the body of the person (moving object) shown in FIG. 21and the reliability on the motion compensating vector is insufficient atpart of the boundary of the person or the background. Utilization ofthis allows detection of the boundary of a moving object. Further, fromthis, a region for which detection of a motion compensating vector mustbe performed preferentially (a region indicated by a thick line in theexample of FIG. 23) when re-detection of a motion compensating vector(re-search of the re-search section 121 hereinafter described withreference to FIG. 29) is to be performed for a decoded image of the MPEG2 and such a policy that, when detection of a motion compensating vectoris performed, for example, a portion having a high reliability is leftas it is whereas, for another portion having a low reliability, a motioncompensating vector is determined once more can be recognized.

In contrast, if a reliability is determined at step S53 and only asignificant motion compensating vector having a high reliability is usedat step S34, then deterioration at a counter portion of an image can besuppressed as hereinafter described with reference to FIG. 46.

FIG. 24 shows another example of configuration of the characteristicamount extraction section 111 of FIG. 3.

In the example of FIG. 24, the characteristic amount extraction section111 includes a re-encoding section 171, an extraction section 172 and anevaluation section 173. The re-encoding section 171 re-encodes the inputsignal supplied from the decoding section 11 or the creation signalsupplied from the signal storage section 33 to acquire re-codedinformation.

The extraction section 172 extracts a motion compensating vectorcorresponding to a candidate vector from within the re-coded informationacquired by the re-encoding section 171. The evaluation section 173compares and evaluates the candidate vector and the motion compensatingvector extracted by the extraction section 172 to determine thereliability of the candidate vector and outputs the determinedreliability to the reliability evaluation section 112.

The reliability calculation process executed by the characteristicamount extraction section 111 of FIG. 24 is described in more detailwith reference to FIG. 25.

As shown in FIG. 25, an encoding section not shown encodes image data ofan original picture to produce a bit stream A and coded information(side information) including motion compensating vectors. Then, thevariable length decoding section 20 (FIG. 1) decodes the bit stream A toproduce a decoded picture A. The re-encoding section 171 re-encodes thedecoded picture (input signal) A supplied from the variable lengthdecoding section 20 to produce a bit stream B and re-coded informationincluding a motion compensating vector B. The bit stream B is decodedinto a decoded picture B by a decoding section not shown.

Accordingly, the extraction section 172 extracts a motion compensatingvector B corresponding to a candidate vector from within the re-encodedinformation acquired by the re-encoding section 171. The evaluationsection 173 compares the motion compensating vector A supplied as acandidate vector from the candidate vector acquisition section 61 andthe motion compensating vector B supplied from the extraction section172 with each other. For example, when the reliability of the motioncompensating vector A is high, even if it is compared with the motioncompensating vector B, the error between them is small. However, if thereliability of the motion compensating vector A is low, the differencebetween the motion compensating vector A and the motion compensatingvector B is great. Accordingly, when the difference between the motioncompensating vector A and the motion compensating vector B is great, theevaluation section 173 can determine that the reliability of the motioncompensating vector A is low. Consequently, the characteristic amountextraction section 122 of FIG. 3 extracts a characteristic amount from amotion compensating vector A determined to have a high reliability anduses the extracted characteristic amount for the tap extraction.

Thus, the motion compensating vector B is used only for thedetermination of the motion compensating vector A by the characteristicamount extraction section 111 of FIG. 24 but is not used for the tapextraction, and only the motion compensating vector A is used for thetap extraction. Accordingly, since the image data processing apparatus 1of FIG. 1 does not utilize a motion compensating vector determined againfrom a creation signal (that is, the motion compensating vector B) toperform time resolution creation but only utilizes a motion compensatingvector of side information (that is, the motion compensating vector A)as an object of comparison to perform a creation process, a timeresolution of a high quality can be achieved.

Subsequently, the reliability calculation process executed by thecharacteristic amount extraction section 111 of FIG. 24 is describedwith reference to a flow chart of FIG. 26. It is to be noted that thisreliability calculation process is another example of the reliabilitycalculation process at step S32 of FIG. 18.

At step S51, the re-encoding section 171 re-encodes the input signalsupplied from the decoding section 11 or the creation signal suppliedfrom the signal storage section 33 to acquire re-encoded information.

At step S52, the extraction section 172 extracts a motion compensatingvector corresponding to the candidate vector from within the re-encodedinformation acquired by the re-encoding section 171 at step S51.

At step S53, the evaluation section 173 compares and evaluates thecandidate vector with the motion compensating vector extracted by theextraction section 172 at step S52 to determine the reliability of thecandidate vector and outputs the reliability to the reliabilityevaluation section 112.

FIG. 27 illustrates a result of comparison between candidate vectors ofa P picture of the original image of FIG. 21 and motion compensatingvectors of re-encoded information corresponding to the candidatevectors. In the example of FIG. 27, it is indicated that a darker blockhas a lower reliability while a whiter block has a higher reliability.It can be seen that many dark blocks are present along the contour ofthe body of the person (moving object) shown in FIG. 21 and thereliability on the motion compensating vector is insufficient at part ofthe boundary of the person or the background.

It is to be noted that, in the reliability determination process at stepS33 of FIG. 18, results in reliability determined by the two reliabilitycalculation processes described above or by different reliabilitycalculation processes hereinafter described may be used in combination(integrally) for the determination or only one of the reliabilities maybe used for the determination.

After the reliability determination process is completed in such amanner as described above, the motion compensating vector selectionsection 102 executes, subsequently at step S23 of FIG. 12, a selectionprocess of a motion compensating vector from among the significantcandidate vectors output from the reliability evaluation section 112.The motion compensating vector selection process is described withreference to a flow chart of FIG. 28.

At step S61, the re-search section 121 of the motion compensating vectorselection section 102 discriminates whether or not a candidate vector tobe processed is present. If it is discriminated that a candidate vectorto be processed is present, then the re-search section 121 uses, at stepS62, the input signal supplied from the decoding section 11 and thecreation signal supplied from the signal storage section 33 to execute are-search process for a motion compensating vector from the significantvectors as seen in FIG. 29.

In the example of FIG. 29, the axis of ordinate indicates the temporaldirection while the axis of abscissa indicates the horizontal directionof frames. The frames (I3, B4, B5, P6) of FIG. 13B are shown in thedisplaying order from above and the intermediate frames (f11, f12, f13)are created between adjacent ones of the frames.

Each arrow mark extending between different frames represents asignificant candidate vector. The candidate vectors include, in orderfrom the left, three candidate vectors, which can be utilized forcreation of the intermediate frame f13 (a candidate vector whose startpoint is the P6 picture and whose end point is the B5 picture, anothercandidate vector whose start point is the P6 picture and whose end pointis the B4 picture and a further candidate vector whose start point isthe I3 picture and whose end point is the P6 picture), three candidatevectors, which can be utilized for creation of the intermediate framef12 (a candidate vector whose start point is the P6 picture and whoseend point is the B4 picture, another candidate vector whose start pointis the I3 picture and whose end point is the B5 picture and a furthercandidate vector whose start point is the I3 picture and whose end pointis the P6 picture), and three candidate vectors, which can be utilizedfor creation of the intermediate frame f11 (a candidate vector whosestart point is the I3 picture and whose end point is the P6 picture,another candidate vector whose start point is the I3 picture and whoseend point is the B5 picture and a further candidate vector whose startpoint is the I3 picture and whose end point is the B4 picture).

Further, in each frame, the position thereof at which it contacts withthe start point of an arrow mark indicating a candidate vector is theposition of a reference pixel while the position thereof at which itcontacts with the end point of an arrow mark indicating a candidatevector is the position of a noticed pixel. Further, a dark round on eachintermediate frame at which it crosses with an arrow mark indicating acandidate vector represents the position of a creation pixel at which itcan be created with the candidate vector.

As seen in FIG. 29, a re-search for a motion compensating vector in theneighborhood p1 of the start point (reference pixel) on the start pointframe of each candidate vector is executed based on the end point of thecandidate vector. Further, a re-search for a motion compensating vectorin the neighborhood p2 of the end point (noticed pixel) on the end pointframe of each candidate vector is executed based on the start point ofthe candidate vector. Furthermore, a re-search for motion compensatingvectors in the neighborhood p1 of the start point and in theneighborhood p2 of the end point of each candidate vector is executedbased on the neighborhood p3 of a midpoint on one of frames, which isnear to the creation pixel, of the candidate vector between the startpoint frame and the end point frame (for example, with regard to thethird leftmost candidate vector of FIG. 29 whose start point is the I3picture and whose end point is the P6 picture, the B4 frame and the B5frame are present. The frame near to the creation pixel [frame f13] isthe frame of the B5 picture).

More particularly, as seen in FIG. 30, in the neighborhood p1 of thestart point, in the neighborhood p2 of the end point or in theneighborhood p3 of the midpoint of a candidate vector, a more adaptivemotion compensating vector is searched for by weighted meaning or thelike taking not only a pixel unit represented by a large circularpattern but also a half-pixel unit represented by a small circularpattern into consideration. In particular, if an optimum motioncompensating vector corresponding to a candidate vector is re-searchedout at a pixel or a half pixel (position between pixels) in theneighborhood p1 of the start point, in the neighborhood p2 of the endpoint or in the neighborhood p3 of the midpoint of the candidate vector,then the motion compensating vector is output as a candidate vector tothe characteristic amount extraction section 122. If such an optimummotion compensating vector is not re-searched out, then the candidatevector supplied from the reliability evaluation section 112 is output asit is to the characteristic amount extraction section 122.

By executing a re-search for a significant candidate vector in such amanner as described above, a more appropriate candidate vector isacquired. As a result, an image of a higher picture quality can beproduced.

Then, at step S63 of FIG. 28, the characteristic amount extractionsection 122 executes a reliability calculation process. This reliabilitycalculation process is similar to the reliability calculation process atstep S32 of FIG. 18 described hereinabove (the process of FIG. 20 orFIG. 26), and description thereof is omitted herein to avoid redundancy.

In particular, in the motion compensating vector selection process, thereliability calculation process is executed again. It is to be notedthat, in this instance, a reliability calculation process same as thereliability calculation process of FIG. 18 (by the characteristic amountextraction section 111 of the reliability determination section 101) maybe performed once again or different reliability calculation processesmay be executed. For example, the reliability calculation process basedon a motion compensation residual (by the characteristic amountextraction section 122 of the motion compensating vector selectionsection 102) is performed in the reliability determination process andthen, in the motion compensating vector selection process, a motioncompensating vector corresponding to a candidate vector by re-encodingis executed and a reliability calculation process by comparison andevaluation is performed (or the processes are executed in the reverseorder), or else both reliability calculation processes may be executedin each case.

For example, in the present case, if a motion compensating vectorcorresponding to a candidate vector by re-encoding is extracted and thereliability calculation process by comparison and evaluation isperformed at step S36 of FIG. 28, then at step S64, the motioncompensating vector decision section 123 selects that one of candidatevectors having the highest reliability as a selected vector of thenoticed pixel.

At step S65, the motion compensating vector decision section 123discriminates whether or not a selected vector is selected successfully.If it is discriminated at step S65 that a selected vector is notselected successfully, that is, if a candidate vector having the highestreliability cannot be found out, then the motion compensating vectordecision section 123 controls the characteristic amount extractionsection 122 to determine a motion compensation residual from thecandidate vector using the input signal and the creation signal at stepS66 and selects that one of candidate vectors exhibiting the lowestmotion compensation residual as a selected vector at step S67.

If it is discriminated at step S61 that a candidate vector is notpresent, if it is discriminated at step S65 that a selected vector isselected successfully or if a selected vector is selected at step S67,then the processing returns to FIG. 12. At step S24 of FIG. 12, themotion compensating vector decision section 123 calculates a shiftamount of the noticed pixel (central tap) based on the selected vector.The calculation of the shift amount is hereinafter described withreference to FIG. 47.

At step S25, the motion compensating vector decision section 123 outputsthe shift amount and the shift amount detection characteristic amountcalculated at step S24 to the classification section 52 and theprediction arithmetic operation section 54. The shift amount detectioncharacteristic amount output at this time is, for example, thereliability of the selected vector. It is to be noted that, where amotion compensation residual with regard to the selected vector has beendetermined by the characteristic amount extraction section 122, also themotion compensation residual is output as the shift amount detectioncharacteristic amount.

Through the processes described above, from among a plurality of motioncompensating vectors, which are present temporally and spatially, themotion compensating vector having the highest reliability is selected asa selected vector of the noticed pixel, and the shift amount of thecentral tap (noticed pixel) is calculated based on the selected vector.

FIG. 31 shows a second example of configuration of the shift informationextraction section 62 of FIG. 2. It is to be noted that elementscorresponding to those in FIG. 3 are denoted by like reference numerals.In particular, the shift information extraction section 62 of FIG. 31 isconfigured basically similarly to that of FIG. 3. However, thereliability determination section 101 of FIG. 31 is different from thereliability determination section 101 of the shift informationextraction section 62 of FIG. 3 in that an input signal from thedecoding section 11 and a creation signal from the signal storagesection 33 are not supplied but only candidate vectors are supplied fromthe candidate vector acquisition section 61.

Accordingly, since the characteristic amount extraction section 111 ofFIG. 31 cannot utilize the input signal and the creation signal forcalculation of the reliability, it compares a candidate vector withmotion compensating vectors in the proximity of the candidate vector todetermine the reliability of the candidate vector.

FIG. 32 shows an example of configuration of the characteristic amountextraction section 111 of FIG. 31.

In the example of FIG. 32, the characteristic amount extraction section111 includes a neighborhood vector extraction section 181 and acomparison section 182. The neighborhood vector extraction section 181extracts neighborhood motion compensating vectors (hereinafter referredto as neighborhood vectors) corresponding to a candidate vector fromamong motion compensating vectors. The comparison section 182 comparesand evaluates the neighborhood vectors extracted by the neighborhoodvector extraction section 181 with the candidate vector to determine thereliability of the candidate vector and outputs the reliability to thereliability evaluation section 112.

A reliability determination process executed by the reliabilitydetermination section 101 of FIG. 31 is similar to the reliabilitydetermination process described hereinabove with reference to FIG. 18except that the reliability calculation process at step S32 isdifferent. Therefore, illustration and description of the reliabilitydetermination process are omitted to avoid redundancy. Accordingly, onlya different example of the reliability calculation process at step S32(FIGS. 20 and 26) of FIG. 18 is described with reference to a flow chartof FIG. 33.

At step S71, the neighborhood vector extraction section 181 extractsneighborhood vectors corresponding to the candidate vector from amongmotion compensating vectors. The neighborhood vectors are a plurality ofmotion compensating vectors existing spatially. More particularly, aplurality of spatially existing motion compensating vectors areextracted from among the motion compensating vectors acquired by thecandidate vector acquisition section 61.

At step S72, the comparison section 182 compares and evaluates thecandidate vector with the neighborhood vectors extracted by theneighborhood vector extraction section 181 at step S71 to determine thereliability of the candidate vector and outputs the reliability to thereliability evaluation section 112.

FIGS. 34 and 35 illustrate results when, using the original image ofFIG. 21, the reliabilities are converted into numerical values usingdifference absolute value sums of candidate vectors of a B picture (FIG.34) and a P picture (FIG. 35) and corresponding neighborhood vectors (acandidate vector exhibiting a smaller difference absolute value sum fromneighborhood vectors has a higher reliability) and mapped to an image.

In the examples of FIGS. 34 and 35, a darker block represents that thecandidate vector is fluctuating and has a lower reliability whencompared with neighborhood blocks. Further, as seen from FIG. 34, in theframe of the B picture, several blocks exhibit some dispersion on theboundary background of the moving object. However, as seen from FIG. 35,in the frame of the P picture, the tendency of the dispersion appearsmore significantly than that of FIG. 34. This arises from the fact that,as seen in FIG. 13B, the frame of the P picture is spaced by a greatertemporal distance from the reference I picture than the frame of the Bpicture.

Since evaluation of a reliability can be obtained also by comparison ofa candidate vector with neighborhood vectors as described above, thedetermination process of whether or not the reliability is high isexecuted at step S33 of FIG. 18 based on a result of the evaluation.

FIG. 36 shows another example of configuration of the characteristicamount extraction section 111 of FIG. 31.

In the example of FIG. 36, the characteristic amount extraction section111 includes a history extraction section 191 and a comparison section192. The history extraction section 191 determines, from among motioncompensating vectors acquired by the candidate vector acquisitionsection 61, a plurality of motion compensating vectors existing in theforward direction or the backward direction in time to extract a vectorhistory. The comparison section 192 compares and evaluates a candidatevector with motion compensating vectors in the past (hereinafterreferred to as past vectors) or motion compensating vectors in thefuture (hereinafter referred to as future vectors) corresponding to thecandidate vector acquired from the vector history to determine thereliability of the candidate vector and outputs the reliability to thereliability evaluation section 112.

The reliability calculation process executed by the characteristicamount extraction section 111 of FIG. 36 is described with reference toa flow chart of FIG. 37.

At step S81, the history extraction section 191 extracts the vectorhistory corresponding to the candidate vector. More particularly, thehistory extraction section 191 determines, from among the motioncompensating vectors acquired by the candidate vector acquisitionsection 61, a plurality of motion compensating vectors existing in theforward direction or the backward direction in time to extract a vectorhistory.

At step S82, the comparison section 192 compares and evaluates, based onthe vector history extracted by the history extraction section 191 atstep S81, the candidate vector with the past vectors or the futurevectors corresponding to the candidate vector to determine thereliability of the candidate vector (for example, a candidate vectorhaving a lower difference absolute value sum from the past vectors (orfuture vectors) has a higher reliability) and outputs the reliability tothe reliability evaluation section 112.

FIG. 38 shows a further example of configuration of the characteristicamount extraction section 111 of FIG. 31.

In the example of FIG. 38, the characteristic amount extraction section111 includes a history extraction section 201 and a discontinuityevaluation section 202. The history extraction section 201 determines,from among motion compensating vectors acquired by the candidate vectoracquisition section 61, a plurality of motion compensating vectorsexisting in the forward direction or the backward direction in time toextract a vector history. The discontinuity evaluation section 202evaluates the discontinuity in motion corresponding to the candidatevector acquired from the vector history to determine the reliability ofthe candidate vector and outputs the reliability to the reliabilityevaluation section 112.

The reliability calculation process executed by the characteristicamount extraction section 111 of FIG. 38 is described with reference toa flow chart of FIG. 39.

At step S91, the history extraction section 201 extracts a vectorhistory corresponding to the candidate vector. At step S92, thediscontinuity evaluation section 202 evaluates the discontinuity inmotion corresponding to the candidate vector acquired from the vectorhistory at step S91 to determine the reliability of the candidate vector(for example, the higher the continuity, the higher the reliability) andoutputs the reliability to the reliability evaluation section 112.

The processes of FIG. 36 (FIG. 37) and FIG. 38 (FIG. 39) described aboveare described in more detail with reference to FIG. 40. In the exampleof FIG. 40, the axis of ordinate indicates the temporal direction, andthe time elapses downwardly in FIG. 40 like I3, B4, B5, P6, . . . .Meanwhile, the axis of abscissa indicates the horizontal direction offrames.

In the example, at an arbitrary one of 12 pixels on the B4 frame, abidirectional vector (a vector history) composed of a forward (past)vector (a motion compensating vector whose start point is the I3 pictureand whose end point is the B4 picture) and a reverse (future) vector (amotion compensating vector whose start point is the P6 picture and whoseend point is the B4 picture) is indicated.

In the example of FIG. 40, if a notice is taken of the third leftmostreverse vector, it can be seen that only this third leftmost reversevector crosses with another reverse vector (has a much differentdirection) and is a fluctuating vector.

By taking notice of the tendency of bidirectional vectors correspondingto a candidate vector and comparing the same with forward vectors orreverse vectors (past vectors or future vectors) as described above, adispersing candidate vector can be detected. Further, by taking noticeof the tendency of bidirectional vectors corresponding to a candidatevector and evaluating the discontinuity in motion, a variation point ofthe candidate vector can be detected.

FIG. 41 shows a still further example of configuration of thecharacteristic amount extraction section 111 of FIG. 31.

In the example of FIG. 41, the characteristic amount extraction section111 includes an extraction section 211 and a comparison section 212. Theextraction section 211 extracts, from among motion compensating vectorsof coded information supplied from the candidate vector acquisitionsection 61, a motion compensating vector over a full screen (hereinafterreferred to as full screen vector). The comparison section 212 comparesand evaluates the candidate vector with the full screen vector extractedby the extraction section 211 to determine a reliability of thecandidate vector and outputs the reliability to the reliabilityevaluation section 112.

The reliability calculation process executed by the characteristicamount extraction section 111 of FIG. 41 is described with reference toa flow chart of FIG. 42.

At step S101, the extraction section 211 extracts a full screen vector.In particular, the extraction section 211 arithmetically operates, basedon motion compensating vectors of coded information supplied from thecandidate vector acquisition section 61, for example, a mean value ofall motion compensating vectors on a screen to extract a full screenvector. At step S102, the comparison section 212 compares and evaluatesthe candidate vector with the full screen vector extracted at step S 101to determine the reliability of the candidate vector and outputs thereliability to the reliability evaluation section 112. Also in thisinstance, for example, a candidate vector exhibiting a lower differenceabsolute value sum from the full screen vector is determined to have ahigher reliability.

As described above, the reliability of a candidate vector can bedetermined by comparison and evaluation of the candidate vector withmotion compensating vectors on a full screen. Further, though not shown,in order to determine a dispersion of a candidate vector, it isotherwise possible to determine, without using such neighborhoodvectors, full screen vector or vector history as described above, thereliability regarding a vector at a half pixel and execute thereliability determination process from a statistic or the like based onthe thus determined reliability. Furthermore, while, in the descriptionabove, the reliability determination process involves determination withan evaluation value determined by a single calculation process, it mayotherwise involve determination from a statistic of a combination ofevaluation values determined by the reliability calculation process.

FIG. 43 shows a third example of configuration of the shift informationextraction section 62 of FIG. 2. It is to be noted that those elements,which correspond to those of FIG. 3, are denoted by like referencenumerals. In particular, the shift information extraction section 62 ofFIG. 32 is configured basically similarly to that of FIG. 3. However,the motion compensating vector selection section 210 in the example ofFIG. 43 is different from the motion compensating vector selectionsection 102 of the shift information extraction section 62 of FIG. 43 inthat an input signal and a creation signal are not input thereto fromthe decoding section 11 and the signal storage section 33 but onlysignificant candidate vectors and reliabilities of them are inputthereto from the reliability determination section 101.

In particular, the motion compensating vector selection section 210compares significant candidate vectors with the other motioncompensating vectors corresponding to the candidate vectors (forexample, neighborhood vectors or past vectors) to select one of thecandidate vectors having the highest reliability as a selected vector.

The motion compensating vector selection process executed by the motioncompensating vector selection section 210 of FIG. 43 is described withreference to a flow chart of FIG. 44. It is to be noted that processesat steps S111 to S113 of FIG. 44 are similar to those at steps S61, S63and S64 of FIG. 28 (the process of FIG. 44 is similar to the motioncompensating vector selection process of FIG. 28 except that the stepsS62, S65 and S67 of FIG. 28 are excluded and that the reliabilitycalculation process at step S112 is different from that at step S63).Further, the reliability calculation process at step S112 of FIG. 44 issimilar to the reliability calculation processes described hereinabovewith reference to FIGS. 33, 37, 39 and 42, and description of the sameis omitted herein to avoid redundancy.

In particular, since only the significant candidate vectors from thereliability determination section 101 are input to the motioncompensating vector selection section 210 but the input signal and thecreation signal are not input as described hereinabove, the re-searchfor a candidate vector executed using the input signal and the creationsignal (at step S62 of FIG. 28) and the reliability calculation processincluding calculation of a motion compensation residual and so forth(FIG. 20 or 26) are not performed. Thus, in the reliability calculationprocess at step S112, the motion compensating vector selection section210 executes the reliability calculation processes of FIGS. 33, 37, 39and 42 in which the input signal and the creation signal are not usedbut the candidate vectors are used for evaluation through comparisonwith the other motion compensating vectors (for example, neighborhoodvectors or past vectors) corresponding to the candidate vectors. Then atstep S113, one of the candidate vectors having the highest reliabilityis selected as a selected vector, whereafter the motion compensatingvector selection process is ended.

It is to be noted that, since the motion compensating vector selectionprocess of FIG. 44 does not involve utilization of the input signal andthe creation signal, it does not include the processes at steps S65 toS67 of FIG. 28. Accordingly, if a candidate vector (motion compensatingvector) having the highest reliability cannot be found out from amongthe candidate vectors at step S113, then neither a shift amount nor ashift amount detection characteristic amount is calculated at steps S24and S25 of FIG. 12. Thus, as a later process when a candidate vectorhaving the highest reliability cannot be found out from among thecandidate vectors at step S113, a process similar to the creationprocess (processes at the steps beginning with step S202) executed in anintra-frame process (a process in which any other picture is notreferred to [a process in which no motion compensation predictionprocess is executed]) which is hereinafter described with reference to aflow chart of FIG. 59 is executed.

It is to be noted that the combination of the reliability calculationprocesses executed in the reliability determination process and themotion compensating vector selection process may be repetitive use of asimilar process or a combination of different processes. Further, anynumber of such processes may be used for the combination.

FIG. 45 illustrates an image of an intermediate frame obtained by usingthe original image of FIG. 21 and using results of the comparison andevaluation (FIG. 33) of candidate vectors and neighborhood vectors andresults of the comparison and evaluation (FIG. 26) of the candidatevectors and motion compensating vectors from re-encoded informationobtained by re-encoding the input signal from within the reliabilitycalculation process executed in such a manner as described above toselect those candidate vectors having low evaluation values as selectedvectors to execute creation of a time resolution.

It can be seen that, in the example of FIG. 45, failure of pixelsparticularly on the boundary of the person (moving object) is suppressedand the image is improved considerably when compared with the example ofFIG. 22.

Meanwhile, FIG. 46 illustrates a result when, using the original imageof FIG. 21, reliability evaluation similar to that of the example ofFIG. 45 is executed to select candidate vectors each having a highreliability as selected vectors and an image of an intermediate frame isproduced based on the selected vectors and then the reliabilities of theselected vectors are converted into numerical values and mapped to theimage of the intermediate frame.

In the example of FIG. 46, it is shown that the thicker and darker thecontour, the lower the reliability. It can be seen that the counter ofthe body of the person is thinner and lighter. At part of the boundaryof the person (moving object) or the background, the reliability of themotion compensating vector is higher than that where those candidatevectors with which the motion compensation residual exhibits the lowestvalue are selected as selected vectors to produce an image of anintermediate frame (FIG. 23). Consequently, it can be recognized thatthe motion compensating vectors are considerably stable.

Since reliability evaluation of temporal and spatial motion compensatingvectors is executed and only significant motion compensating vectors areselected in such a manner as described above, the picture quality of anintermediate frame created is improved.

Further, not only a shift amount of a central tap (noticed pixel)corresponding to a selected vector selected by the motion compensatingvector decision section 123 but also a shift amount detectioncharacteristic amount (for example, the reliability or motioncompensation residual of the motion compensating vector) is output tothe classification section 52 and the prediction arithmetic operationsection 54. Since the shift amount detection characteristic representsreliability evaluation of the motion compensating vector itself includedin the input signal, a region to which attention should be paid when thedetermined input signal is processed or a region which is likely to betaken in error when the boundary of a moving body or a motioncompensating vector is to be detected again can be indicated. Thus, itis possible to execute a classification process hereinafter describedbased on the indicated region or boundary.

A shift amount is calculated in such a manner as described above at stepS2 of FIG. 11. Processes at the steps beginning with step S3 of FIG. 11,which are executed using the shift amount calculated as described above,are described in more detail below.

At step S3, the class tap construction section 131 of the classificationsection 52 receives the input signal from the decoding section 11 andthe creation signal from the signal storage section 33 as inputs theretoand determines a creation pixel by such tap extraction as illustrated inFIG. 47 based on the shift amount and the shift amount detectioncharacteristic amount supplied from the motion compensating vectordecision section 123. Then, the class tap construction section 131constructs a class tap in accordance with the creation pixel and outputsthe class tap to the class code production section 132.

An example of tap extraction where the tap structure includes 3×3 pixelsis described with reference to FIGS. 47A-B. FIG. 47A illustrates tapconstruction on a plane where the axis of ordinate is the verticaldirection of a frame and the axis of abscissa is the temporal direction.FIG. 47B shows tap construction on a plane where the axis of ordinate isthe vertical direction of a frame and the axis of abscissa is thehorizontal direction of the frame.

In the example of FIGS. 47A-B, creation of an intermediate frame F2between a past frame F1 and a future frame F3 on the time axis isillustrated. It is to be noted that a indicated on the time axisrepresents a value for internal division of a motion compensating vectorMV4 determined from a distance between and/or across frames or the like.

In the present example, the past frame F1 is a frame of a motioncompensation reference source and is constructed from the creationsignal. Meanwhile, the future frame F3 is a frame after motioncompensation reference and is constructed from the input signal. Then,the motion compensating vector Mv4 whose end point is a noticed pixel a4on the future frame F3 and whose start point is a reference pixel b4 onthe past frame F1 is selected. Based on the motion compensating vectorMv4, a shift amount S1 from the reference pixel (central tap) b4 on thepast frame F1 and another shift amount S2 from the noticed pixel(central tap) a4 on the future frame F3 are determined. It is to benoted that the shift amount S1 and the shift amount S2 are calculatedand supplied by the motion compensating vector decision section 123 atstep S24 of FIG. 12 described hereinabove.

In particular, a creation pixel c4 of the intermediate frame F2corresponds in position to a pixel b0 of the past frame F1 and anotherpixel a0 of the future frame F3. However, since a better image isobtained if creation of a time resolution is performed based on themotion compensating vector MV4, as the reference source on the pastframe F1, the pixel b4 is used in place of the pixel b0. This can beconsidered equivalent to shifting (tap extraction) of the pixel of thereference source from the position of the pixel b4 to the position ofthe pixel b0 by the shift amount S1 to utilize the pixel of thereference source for creation of a time resolution (creation of anintermediate frame). Similarly, as a reference destination on the futureframe F3, the pixel a4 is used in place of the pixel a0. This can beconsidered equivalent to shifting (tap extraction) of the pixel of thereference destination from the position of the pixel a4 to the positionof the pixel a0 by the shift amount S2 to utilize the pixel of thereference source for creation of a time resolution (creation of anintermediate frame). Thus, such shifting is only conceptual, notactually performed.

It is to be noted that, since shifting (tap extraction) is performed ina unit of a tap group, a tap group AE4 and another tap group BE4 eachincluding 3×3 pixels in the neighborhood of the noticed pixel a4 and thereference pixel b4 including the same are determined based on thenoticed pixel a4 and the reference pixel b4, respectively.

In this manner, the tap group AE4 or the tap group BE4 is shiftedconceptively based on the shift amounts S1 and S2 to construct a tapgroup CE4 including the creation pixel c4 on the intermediate frame F2(the center of the tap group CE4). More particularly, the tap group CE4is determined based on the sum of values of the tap group BE4 weightedwith an internal division value a and values of the tap group AE4weighted with another internal division value (1−α).

It is to be noted that, in a B picture or a P picture of the MPEG 2, thepositional relationship of the reference source and the referencedestination and the temporal positional relationship of them maypossibly be different from each other.

A class tap or taps (pixel or pixels) corresponding to the creationpixel c4 are constructed from within the tap group CE4 determined insuch a manner as described above. The number and the position of suchclass pixels are determined suitably accounting for memory limitations,processing speed and such.

For example, the class tap construction section 131 constructs, based onthe shift amount and the shift amount detection characteristic amount,10 taps including 5 taps (pixels) from the input signal and 5 taps fromthe creation signal, which is positioned preceding in time (in the past)with respect to the input signal, as class taps (pixels in theneighborhood of the creation pixel).

In FIGS. 48A-B, which shows an example of configuration of class taps inthe input signal and the creation signal, FIG. 48A shows class taps on aplane whose axis of ordinate indicates the vertical direction of a frameand whose axis of abscissa indicates the temporal direction. Meanwhile,FIG. 48B shows class taps on another plane whose axis of ordinateindicates the vertical direction of the frame and whose axis of abscissaindicates the horizontal direction of the frame.

In the example of FIG. 48A, the preceding tap group (group on the leftside in the figure) in the temporal direction is a class tap group ofthe creation signal, and the succeeding tap group (group on the rightside in the figure) is a class tap group of the input signal. Each ofthe class tap groups of the creation signal and the input signalincludes five class taps as seen in FIG. 48B. In other words, while eachof the class tap groups is shown including only three class taps in FIG.48A because some of them overlap with each other, it actually includesfive class taps.

Thus, at step S4, the class code production section 132 extracts acharacteristic amount of each of the class taps constructed by the classtap construction section 131 based on the shift amount and the shiftamount detection characteristic amount from the motion compensatingvector decision section 123. The characteristic amount may be, forexample, a shift amount (magnitude or direction) of a tap, a motioncompensation residual from within the shift amount detectioncharacteristic amount, the reliability of the motion compensating vectorfrom within the shift amount detection characteristic amount, a patternof a block boundary upon the tap extraction or the like.

A pattern of a block boundary is described with reference to FIGS.49A-B. The position of a boundary of a block composed of 8×8 pixelsassumes one of 64 patterns as a result of the tap extraction describedhereinabove with reference to FIGS. 47A-B. FIG. 49A shows the patternsof a block boundary upon the tap extraction. Patterns of a tap drawnnear and a block boundary of a block when the tap extraction isperformed are divided into 64 patterns as shown in FIG. 49A.

In the example of FIG. 49A, each of lines extending in a horizontaldirection and lines extending in a vertical direction represents aboundary of an original block (prior to the tap extraction). The pattern0 is a pattern in a case wherein the boundary of the block of the drawnnear tap coincides with the boundary of the original block; the pattern1 is a pattern wherein the block of the drawn near tap is positioned onthe left side by a one-pixel distance from the original block; and thepattern 7 is a pattern wherein the block of the drawn near tap ispositioned on the left side by a seven-pixel distance from the originalblock. Further, the pattern 8 is a pattern wherein the block of thedrawn near tap is positioned on the upper side by a one-pixel distancefrom the original block; and the pattern 15 is a pattern wherein theblock of the drawn near tap is positioned on the upper side by aone-pixel distance from the original block and on the left side by aseven-pixel distance from the original block. Furthermore, the pattern56 is a pattern wherein the block of the drawn near tap is positioned onthe upper side by a seven-pixel distance from the original block; andthe pattern 63 is a pattern wherein the block of the drawn near tap ispositioned on the upper side by a seven-pixel distance from the originalblock and on the left side by a seven-pixel distance from the originalblock. It is to be noted that, since also the patterns 2 to 6, 9 to 14,16 to 55, and 56 to 62 can be explained similarly, description thereofis omitted herein.

In FIG. 49B, the axis of abscissa indicates the positions (patternnumbers) of the block boundaries of the patterns on the block boundariespatterned as seen in FIG. 49A, and the axis of ordinate indicates errorsbetween the original image and a decoded image obtained by encoding theoriginal image and then decoding the encoded image. It can be seen fromFIG. 49B that, if the drawn near tap is on a pattern PA1 wherein it ispositioned in the proximity of the original block boundary (for example,the pattern 0, 7, 15, 56 or 63), the error between the original imageand the decoded image is great. In contrast, it can be seen that thereis a tendency that, where the position of the original block boundary ison a pattern PA2 wherein it is positioned in the inside of the block(for example, one of the patterns 24 to 53), the error between theoriginal image and the decoded image is small.

Since block noise is generated on a block boundary, if the drawn neartap is included in an end portion of a block boundary, a difference inluminance value occurs at the end of the block. An end edge of a blockincludes, where it is represented with DCT coefficients, many highfrequency components, and such high frequency components decreasethrough quantization with a high probability, which is likely to causean error. Therefore, those patterns in which the drawn near tap ispositioned in the inside of a block boundary can decrease errors betweenthe original image and the decoded image.

Accordingly, if a pattern of a block boundary upon the tap extraction isdetermined as a characteristic amount and the block is classified basedon the characteristic amount, then coding distortion such as blockdistortion or “mosquito noise” is suppressed and an intermediate framehaving a good quality in that the original image and the decoded imageexhibit minimized errors is created.

At step S5 of FIG. 11, the class code production section 132 determinesa class of the tap (pixel) based on a threshold value or the like set inadvance in accordance with the extracted characteristic amount of theclass tap, produces such a class code as seen in FIG. 50 and outputs theclass code to the prediction coefficient selection section 53.

When to perform classification by the class taps, it is possible to usea bit string obtained by arranging a bit string representative of samplevalues of data, which form the class taps, as they are in apredetermined order as a class code. In this instance, however, thenumber of classes (total number of classes) becomes very great.Therefore, for the classification, a compression process such as, forexample, a K-bit ADRC (Adaptive Dynamic Range Coding) process isadopted. For example, when a 1-bit ADRC process is adopted (where K=1),the sample value of data, which form the class tap, is formed from 1bit.

Accordingly, in the class code illustrated in FIG. 50, classescorresponding to the 64 patterns of block boundaries are represented by6 bits of R1. However, if the PA2 exhibiting a small error are not used,but only the patterns PA1 exhibiting a great error are used, then thepatterns of block boundaries can be represented with a smaller number ofbits (for example, 4 bits).

Further, classes corresponding to motion compensation residuals arerepresented with 2 bits of R2, and classes corresponding to shiftamounts are represented with 4 bits of R3. Further, 1-bit ADRC codes for10 taps, which form a class tap, are represented with 10 bits of R4. Asa result, in the example of FIG. 50, a class code is represented withtotaling 22 bits.

Then at step S6 of FIG. 11, the prediction coefficient selection section53 selects the coefficient memory from 71-0 to 71-N. The selectedcoefficient memory corresponds to the parameter B supplied from theparameter control section 27 of the decoding section 11. The predictioncoefficient selection section 53 selects, from among the predictioncoefficient data stored in advance in the selected coefficient memory,the prediction coefficient data corresponding to the class code producedby the classification section 52, and outputs the selected predictioncoefficient data to the arithmetic operation section 142 of theprediction arithmetic operation section 54. It is to be noted that theprediction coefficient data stored in the coefficient memories 71-0 to71-N in accordance with the parameter B have been calculated by alearning apparatus 301, which is hereinafter described with reference toFIG. 51.

At step S7, the prediction tap construction section 141 of theprediction arithmetic operation section 54 receives the input signalfrom the decoding section 11 and the creation signal stored in thesignal storage section 33 as inputs thereto, determines a creation pixelby the tap extraction described hereinabove with reference to FIG. 47based on the shift amount and the shift amount detection characteristicamount supplied from the motion compensating vector decision section123, constructs a prediction tap in accordance with the creation pixeland outputs the prediction tap to the arithmetic operation section 142.

The arithmetic operation section 142 performs, at step S8, a predictionarithmetic operation process using the prediction coefficient dataselected by the prediction coefficient selection section 53 for theprediction tap constructed in such a manner as described above toproduce a creation signal of a created time resolution and outputs thecreation signal.

The creation signal is output to a monitor or the like, which forms anoutput section 417 (FIG. 62), and is stored into the signal storagesection 33 so as to be used for production of a next creation signal.

As described above, a plurality of motion compensating vectors areextracted from within coded information transmitted and stored as sideinformation. Significant motion compensating vectors are selected basedon the extracted motion compensating vectors. Then, an intermediateframe is created based on the significant motion compensating vectorsand shift amount detection characteristic amounts such as thereliability. As a result, coding distortion is suppressed and the imagequality is improved.

FIG. 51 shows a first example of configuration of a learning apparatus301 for learning prediction coefficient data for different classes to bestored into the coefficient memories 71-0 to 71-N built in theprediction coefficient selection section 53 of the classificationadaptation processing sections 31 of FIG. 2.

It is to be noted that the learning apparatus 301 may be included in theimage data processing apparatus 1 of FIG. 1, which has theclassification adaptation processing sections 31, or may otherwise beformed as an independent apparatus.

An input signal (this input signal is different from the input signaloutput from the picture selection section 23 of the decoding section 11of FIG. 1) as a teacher signal is input to an encoding section 311 andis further input as a teacher signal to a prediction coefficientcalculation section 315. Also a parameter B (Volume value) is input tothe learning apparatus 301 from a parameter production section notshown, which is similar to the parameter control section 27 of FIG. 1.

The encoding section 311 encodes the input signal and outputs theencoded signal to a decoding section 312. The decoding section 312decodes the encoded signal in accordance with the parameter B andoutputs the decoded signal as a first student signal to a shift amountarithmetic operation section 313, a classification section 314 and theprediction coefficient calculation section 315.

A creation signal produced from the input signal as a student signal bya creation section not shown similar to the creation section 12 of FIG.1 is input as a second student signal to the shift amount arithmeticoperation section 313, classification section 314 and predictioncoefficient calculation section 315.

The shift amount arithmetic operation section 313 is configuredbasically similarly to the shift amount arithmetic operation section 51of FIG. 2. The shift amount arithmetic operation section 313 acquirescandidate vectors from among motion compensating vectors included incoded information supplied from the decoding section 312. The shiftamount arithmetic operation section 313 evaluates the reliabilities ofthe candidate vectors using the first student signal and the secondstudent signal to select that one of the candidate vectors, which hasthe highest reliability, as a selected vector. Further, the shift amountarithmetic operation section 313 determines a shift amount of a centraltap (noticed pixel) and a shift amount detection characteristic amountsuch as a reliability or a motion compensation residual of the motioncompensating vector based on the selected vector. The shift amountarithmetic operation section 313 outputs the shift amount and the shiftamount detection characteristic amount to the classification section 314and the prediction coefficient calculation section 315.

The classification section 314 is configured basically similarly to theclassification section 52 of FIG. 2, and uses the first student signaland the second student signal to determine a creation pixel by the tapextraction based on the shift amount of the central tap and the shiftamount detection characteristic amount supplied from the shift amountextraction section 313, constructs a class tap of the creation pixel,and outputs a class code to the prediction coefficient calculationsection 315.

The prediction coefficient calculation section 315 uses the firststudent signal and the second student signal to determine a creationpixel by the tap extraction based on the shift amount of the central tapand the shift amount detection characteristic amount supplied from theshift information extraction section 62. The prediction coefficientcalculation section 315 constructs a prediction tap of the creationpixel, uses the prediction tap and the teacher signal (input signal)corresponding to the prediction tap to learn a relationship between theteacher signal and the student signals based on the class code from theclassification section 314 and predicts a prediction coefficient fromthe parameter B to arithmetically operate and produce predictioncoefficient data for each class.

More particularly, for example, a prediction value E[y] of a pixel valuey of a pixel of the teacher signal corresponding to a creation signal(student pixel) determined from the first student signal and the secondstudent signal (the two student signals are hereinafter referred tocollectively and simply as student signals) is determined using a linearprimary combination model defined by a liner combination of a set ofseveral student pixels x₁, x₂, and predetermined prediction coefficientsw₁, w₂, . . . . In this instance, the prediction value E[y] can berepresented by the following expression:E[y]=w ₁ x ₁ +w ₂ x ₂+  (1)

In order to generalize the expression (1), if a matrix W that is a setof prediction coefficients w_(j), a matrix X that is a set of studentsignals x_(ij) and a matrix Y′ that is a set of prediction valuesE[y_(j)] are defined as [Expression  1] $X = \begin{bmatrix}x_{11} & x_{12} & \ldots & x_{1\quad J} \\x_{21} & x_{22} & \ldots & x_{2\quad J} \\\ldots & \ldots & \ldots & \ldots \\x_{I\quad 1} & x_{I\quad 2} & \ldots & x_{IJ}\end{bmatrix}$ ${W = \begin{bmatrix}w_{1} \\w_{2} \\\ldots \\w_{J}\end{bmatrix}},{Y^{\prime} = \begin{bmatrix}{E\left\lbrack y_{1} \right\rbrack} \\{E\left\lbrack y_{2} \right\rbrack} \\\ldots \\{E\left\lbrack y_{J} \right\rbrack}\end{bmatrix}}$then, the following observation equation is established:XW=Y′  (2)

Here, the component x_(ij) of the matrix X signifies the “j”th studentsignal in the “i”th set of student signals (a set of student signals tobe used for prediction of the “i”th student signal Y_(i)) (predictioncoefficients), and the component w_(j) of the matrix W represents the“j”th student signal in the set of student signals. Accordingly,E[y_(i)] represents a prediction value of the “i”th teacher signal. Itis to be noted that y on the left side of the expression (1) representsthe component y_(i) of the matrix Y from which the suffix i is omitted,and also x₁, x₂, . . . on the right side of the expression (1) representthe components x_(ij) of the matrix X from which the suffix i isomitted.

It is examined here to apply, for example, the least square method tothe observation equation of the expression (2) to determine theprediction value E[y] proximate to the original pixel value y. In thisinstance, if the matrix Y, which is a set of true pixel values y makinga teacher signal and the matrix E, which is a set of residuals e of theprediction values E[y] for the pixel values y, are defined by$\left\lbrack {{Expression}\quad 2} \right\rbrack\begin{matrix}{{E = \begin{bmatrix}e_{1} \\e_{2} \\\ldots \\e_{I}\end{bmatrix}},{Y = \begin{bmatrix}y_{1} \\y_{2} \\\ldots \\y_{I}\end{bmatrix}}} & \quad\end{matrix}$then the following residual expression is established from theexpression (2):XW=Y+E   (3)

In this instance, the prediction coefficient w_(j) for determining theprediction value E[y] proximate to the original pixel value y can bedetermined by minimizing the square error$\left\lbrack {{Expression}\quad 3} \right\rbrack\begin{matrix}{\sum\limits_{i = 1}^{I}e_{i}^{2}} & \quad\end{matrix}$

Accordingly, where a differentiation value of the square error givenabove with respect to the prediction coefficient w_(j) is 0, that is,where the prediction coefficient w_(j) satisfies the followingexpression, it has an optimum value for determination of the predictionvalue E[y] proximate to the original pixel value y. $\begin{matrix}\left\lbrack {{Expression}\quad 4} \right\rbrack & \quad \\{{{e_{1}\frac{\partial e_{1}}{\partial w_{j}}} + {e_{2}\frac{\partial e_{2}}{\partial w_{j}}} + \ldots + {e_{I}\frac{\partial e_{I}}{\partial w_{j}}}} = {0\left( {{j = 1},2,\ldots\quad,J} \right)}} & (4)\end{matrix}$

Thus, the following expression is established by differentiating theexpression (3) with respect to the prediction coefficient w_(j):$\begin{matrix}\left\lbrack {{Expression}\quad 5} \right\rbrack & \quad \\{{\frac{\partial e_{i}}{\partial w_{1}} = x_{i\quad 1}},{\frac{\partial e_{i}}{\partial w_{2}} = x_{i\quad 2}},\ldots\quad,{\frac{\partial e_{i}}{\partial w_{J}} = x_{iJ}},\left( {{i = 1},2,\ldots\quad,I} \right)} & (5)\end{matrix}$

From the expressions (4) and (5), the expression (6) is obtained:$\begin{matrix}\left\lbrack {{Expression}\quad 6} \right\rbrack & \quad \\{{{\sum\limits_{i = 1}^{I}{e_{i}x_{i\quad 1}}} = 0},{{\sum\limits_{i = 1}^{I}{e_{i}x_{i\quad 2}}} = 0},\ldots\quad,{{\sum\limits_{i = 1}^{I}{e_{i}x_{i\quad J}}} = 0}} & (6)\end{matrix}$

Further, by taking the relationship of the residual e_(i) intoconsideration with the student signal x_(ij), prediction coefficientw_(j) and teacher signal y_(i) of the residual equation of theexpression (3), the following normal equation can be obtained from theexpression (6): $\begin{matrix}\left\lbrack {{Expression}\quad 7} \right\rbrack & \quad \\\left\{ \begin{matrix}{{{\left( {\sum\limits_{i = 1}^{I}{x_{i\quad 1}x_{i\quad 1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{I}{x_{i\quad 1}x_{i\quad 2}}} \right)w_{2}} + \ldots + {\left( {\sum\limits_{i = 1}^{I}{x_{i\quad 1}x_{ij}}} \right)w_{J}}} = \left( {\sum\limits_{i = 1}^{I}{x_{i\quad 1}y_{i}}} \right)} \\{{{\left( {\sum\limits_{i = 1}^{I}{x_{i\quad 2}x_{i\quad 1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{I}{x_{i\quad 2}x_{i\quad 2}}} \right)w_{2}} + \ldots + {\left( {\sum\limits_{i = 1}^{I}{x_{i\quad 2}x_{iJ}}} \right)w_{J}}} = \left( {\sum\limits_{i = 1}^{I}{x_{i\quad 2}y_{i}}} \right)} \\\ldots \\{{{\left( {\sum\limits_{i = 1}^{I}{x_{i\quad J}x_{i\quad 1}}} \right)w_{1}} + {\left( {\sum\limits_{i = 1}^{I}{x_{i\quad J}x_{i\quad 2}}} \right)w_{2}} + \ldots + {\left( {\sum\limits_{i = 1}^{I}{x_{i\quad J}x_{iJ}}} \right)w_{J}}} = \left( {\sum\limits_{i = 1}^{I}{x_{i\quad J}y_{i}}} \right)}\end{matrix} \right. & (7)\end{matrix}$

It is to be noted that, if the matrix (covariance matrix) A and thevector v are defined as $\begin{matrix}{\left\lbrack {{Expression}\quad 8} \right\rbrack{A = \begin{pmatrix}{\sum\limits_{i = 1}^{I}{x_{i\quad 1}x_{i\quad 1}}} & {\sum\limits_{i = 1}^{I}{x_{i\quad 1}x_{i\quad 2}}} & \ldots & {\sum\limits_{i = 1}^{I}{x_{i\quad 1}x_{i\quad J}}} \\{\sum\limits_{i = 1}^{I}{x_{i\quad 2}x_{i\quad 1}}} & {\sum\limits_{i = 1}^{I}{x_{i\quad 2}x_{i\quad 2}}} & \ldots & {\sum\limits_{i = 1}^{I}{x_{i\quad 2}x_{i\quad J}}} \\\quad & \quad & \ldots & \quad \\{\sum\limits_{i = 1}^{I}{x_{i\quad J}x_{i\quad 1}}} & {\sum\limits_{i = 1}^{I}{x_{i\quad J}x_{i\quad 2}}} & \ldots & {\sum\limits_{i = 1}^{I}{x_{i\quad J}x_{i\quad J}}}\end{pmatrix}}{v = \begin{pmatrix}{\sum\limits_{i = 1}^{I}{x_{i\quad 1}y_{i}}} \\{\sum\limits_{i = 1}^{I}{x_{i\quad 2}y_{i}}} \\\vdots \\{\sum\limits_{i = 1}^{I}{x_{i\quad J}y_{i}}}\end{pmatrix}}} & \quad\end{matrix}$and the vector W is defined as represented by the Expression 1, then thenormal equation given by the expression (7) can be represented by theexpressionAW=v   (8)

A number of such normal equations of the expression (7) equal to thenumber J of the prediction coefficients w_(j) to be determined can beestablished by preparing a certain number of sets of the student signalsx_(ij) and the teacher signal y_(i). Accordingly, by solving theexpression (8) with respect to the vector W (it is to be noted that, inorder to solve the expression (8), the matrix A of the expression (8)must be a singular matrix), an optimum prediction coefficient (here, aprediction coefficient minimizing the square error) w_(j) can bedetermined. It is to be noted that, in order to solve the expression(8), it is possible to use, for example, a brushing method (theGauss-Jordan elimination method) and so forth.

As described above, by determining an optimum prediction coefficient,that is, a prediction coefficient w_(j), which minimizes a statisticerror of a prediction value of a pixel value, a prediction value E[y]proximate to the original pixel value y can be determined from theexpression (1) using the determined prediction coefficient w_(j).

The prediction coefficient data produced in such a manner as describedabove is stored into one of the coefficient memories 316-0 to 316-N inaccordance with the parameter B of the same. For example, predictioncoefficient data learned from a decoded signal where the parameter B is“1.00” and the bit rate is 10 Mbps is stored into the coefficient memory316-9. Prediction coefficient data learned from a decoded signal wherethe parameter B is “0.90” and the bit rate is 9 Mbps is stored into thecoefficient memory 316-8. Prediction coefficient data learned from adecoded signal where the parameter B is “0.10” and the bit rate is 1Mbps is stored into the coefficient memory 316-0.

As described above, in the learning apparatus 301, a plurality ofprediction coefficient data are produced from a plurality of teachersignals and a plurality of student signals in accordance with theparameter B and stored into different memories (selected ones of thecoefficient memories 316-0 to 316-N) in accordance with the parameter B.In other words, the parameter B is a parameter for production of aprediction coefficient.

Subsequently, a learning process of the learning apparatus 301 isdescribed with reference to a flow chart of FIG. 52.

At step S121, the encoding section 311 encodes the input signal as ateacher signal and outputs the encoded signal to the decoding section312. At this time, the encoding section 311 outputs quantizationcharacteristic information and encoded information, which is producedupon the encoding, as side information together with a quantized DCTcoefficient. At step S122, the decoding section 312 decodes the encodedsignal in accordance with the parameter B to produce a first studentsignal and outputs the first student signal to the shift amountarithmetic operation section 313, classification section 314 andprediction coefficient calculation section 315. Simultaneously, thedecoding section 312 outputs also the coded information acquired fromthe encoded signal to the shift amount arithmetic operation section 313.

At step S123, the shift amount arithmetic operation section 313 executesa shift amount calculation process. The shift amount calculation processis similar to the process of the shift amount arithmetic operationsection 51 of FIG. 2 described hereinabove with reference to FIG. 12,and therefore, overlapping description of it is omitted herein to avoidredundancy. By the shift amount calculation process, motion compensatingvectors included in the coded information supplied from the decodingsection 11 and regarded as candidate vectors are evaluated in regard tothe reliability using one of the first student signal and the secondstudent signal (creation signal). Then, one of the candidate vectors,which exhibits the highest reliability, is selected as a selectedvector. Then, a shift amount of a central tap (noticed pixel) and ashift amount detection characteristic amount such as the reliability ora motion compensation residual of the selected vector are determined bythe shift amount arithmetic operation section 313 and output to theclassification section 314 and the prediction coefficient calculationsection 315.

At step S124, the classification section 314 uses, similarly to theclassification section 52 of FIG. 2, the first and second studentsignals to determine a creation pixel by the tap extraction based on theshift amount of the central tap and the shift amount detectioncharacteristic amount supplied from the shift amount arithmeticoperation section 313 and constructs a class tap of the creation pixel.The classification section 314 extracts, similarly to the classificationsection 52 of FIG. 2, a characteristic amount of the class tap based onthe shift amount and the shift amount detection characteristic amount atstep S125, and produces a class code based on the extractedcharacteristic amount of the class tap and outputs the class code to theprediction coefficient calculation section 315 at step S126.

At step S127, the prediction coefficient calculation section 315 usesthe first student signal and the second student signal to determine acreation pixel by the tap extraction based on the shift amount of thecentral tap and the shift amount detection characteristic amountsupplied from the shift amount arithmetic operation section 313 andconstructs a prediction tap of the creation pixel (this process issimilar to the process of the prediction tap construction section 141 ofFIG. 5 of the prediction arithmetic operation section 54 of FIG. 2). Atstep S128, the prediction coefficient calculation section 315 uses theprediction tap and the corresponding teacher signal to learn therelationship between the teacher signal and the student signals based onthe class code from the classification section 314 and predicts aprediction coefficient from the parameter B to arithmetically operateand produce prediction coefficient data for each class. At step S 129,the prediction coefficient calculation section 315 stores the predictioncoefficient data into that one of the coefficient memories 316-0 to316-N, which corresponds to the parameter B, and then ends itsprocessing.

Prediction coefficient data for the individual classes stored in thecoefficient memories 316-0 to 316-N in accordance with the parameter Bin such a manner as described above are stored into the coefficientmemories 71-0 to 71-N of the prediction coefficient selection section 53of FIG. 2.

FIG. 53 shows a second example of configuration of the classificationadaptation processing section 31 of FIG. 1. It is to be noted that thoseelements that correspond to those of FIG. 2 are denoted by likereference numerals. In particular, the classification adaptationprocessing section 31 of FIG. 53 is configured basically similarly tothat of FIG. 2.

However, in the example of FIG. 53, the input signal from the decodingsection 11 or the creation signal from the signal storage section 33 isnot supplied to the shift information extraction section 62 of the shiftamount arithmetic operation section 51. Thus, the shift informationextraction section 62 of FIG. 53 compares and evaluates candidatevectors supplied from the candidate vector acquisition section 61 withthe other motion compensating vectors to evaluate the reliabilities ofthe candidate vectors. Then, the shift information extraction section 62selects that one of the candidate vectors that exhibits the highestreliability as a selected vector, and produces a shift amount of acentral tap and a shift amount detection characteristic amount such asthe reliability of the selected vector based on the selected vector.Then, the shift information extraction section 62 outputs the shiftamount of the central tap and the shift amount detection characteristicamount to the classification section 52 and the prediction arithmeticoperation section 54.

FIG. 54 shows an example of configuration of the shift informationextraction section 62 of FIG. 53. It is to be noted that the shiftinformation extraction section 62 of FIG. 54 is configured basicallysimilarly to that in FIG. 43. However, in the example of FIG. 54,different from the reliability determination section 101 of the shiftinformation extraction section 62 of FIG. 43, the reliabilitydetermination section 101 does not receive the input signal from thedecoding section 11 and the creation signal from the signal storagesection 33 as inputs thereto but receives only candidate vectors fromthe candidate vector acquisition section 61 as an input thereto. Inparticular, the shift information extraction section 62 of FIG. 54includes the reliability determination section 101 of FIG. 31 and themotion compensating vector selection section 210 of FIG. 43.

Accordingly, the characteristic amount extraction section 111 of thereliability determination section 101 of FIG. 54 compares the candidatevectors with other motion compensating vectors (for example,neighborhood vectors, past vectors or the like) to determine thereliabilities of the candidate vectors. The reliability evaluationsection 112 of the reliability determination section 101 discriminateswhether or not the reliabilities of the candidate vectors determined bythe characteristic amount extraction section 111 are higher than apredetermined reference value. Then, the reliability evaluation section112 outputs only those of the candidate vectors (significant vectors),which are discriminated to have higher reliabilities, to the motioncompensating vector selection section 210 together with thereliabilities.

Further, the input signal and the creation signal are not input to themotion compensating vector selection section 210 either, but only thesignificant candidate vectors and the reliabilities of them from thereliability determination section 101 are input to the motioncompensating vector selection section 210.

Accordingly, the motion compensating vector selection section 210 usesand compares the significant candidate vectors with other motioncompensating vectors (for example, neighborhood vectors, past vectors orthe like) to select that one of the candidate vectors having the highestreliability.

A creation process of the classification adaptation processing section31 of FIG. 53 is described with reference to a flow chart of FIG. 55. Itis to be noted that the creation process of FIG. 55 is basically similarto the classification adaptation process of FIG. 11. In particular, ashift amount calculation process executed at step S152 of FIG. 55 isbasically similar to the shift amount calculation process of FIG. 12.However, the reliability determination process executed at step S22 ofFIG. 12 and the reliability calculation process (step S32 of FIG. 18 andstep S112 of FIG. 40) in the motion compensating vector selectionprocess executed at step S23 are replaced by the reliability calculationprocess that does not use the input signal and the creation signal asdescribed above with reference to FIGS. 33, 37, 39 and 42. In otherwords, the shift information extraction section 62 of FIG. 53 does notexecute the reliability calculation process (FIGS. 20 and 26), whichuses the input signal and the creation signal.

Consequently, the shift amount detection characteristic amount outputtogether with the shift amount in the shift amount calculation processat step S152 of FIG. 55 does not include such information as the motioncompensation residual, which is produced based on the input signal andthe creation signal. Accordingly, a class tap is constructed at stepS153 based on the shift amount detection characteristic information(such as, for example, the reliability of the motion compensatingvector) other than such information as the motion compensation residualproduced based on the input signal and the creation signal. Then at stepS154, a characteristic amount is extracted, and at step S155, a classcode is produced. Then at step S156, a prediction coefficient isselected, and a prediction tap is constructed at step S157, whereafter acreation signal is produced and output at step S158.

It is to be noted that the configuration of the shift informationextraction section 62 of FIGS. 2 and 53 allows four combinations fromamong the reliability determination section 101 and the motioncompensating vector selection sections 102 and 210 (shift informationextraction section 62 of FIGS. 3, 31, 43 and 54) and may have anyconfiguration only if the conditions are satisfied.

FIG. 56 shows an example of configuration of the learning apparatus 301that learns prediction coefficient data to be stored into thecoefficient memories 71-0 to 71-N of the classification adaptationprocessing section 31 of FIG. 53. It is to be noted that like elementsto those of FIG. 51 are denoted by like reference characters. Inparticular, the learning apparatus 301 of FIG. 56 is configuredbasically similarly to that in FIG. 51.

However, the decoded signal or the creation signal is not input to theshift amount arithmetic operation section 313. Thus, the shift amountarithmetic operation section 313 of FIG. 56 only uses candidate vectorssupplied from the decoding section 312 and compares and evaluates themwith other motion compensating vectors to evaluate the reliabilities ofthe candidate vectors. Then, the shift amount arithmetic operationsection 313 selects that one of the candidate vectors having the highestreliability as a selected vector and calculates a shift amount of acentral tap and a shift amount detection characteristic amount such asthe reliability of the selected vector based on the selected vector.Thereafter, the shift amount arithmetic operation section 313 outputsthe shift amount and the shift amount detection characteristic amount tothe classification section 314 and the prediction coefficientcalculation section 315.

A learning process of the learning apparatus 301 of FIG. 56 is describedwith reference to a flow chart of FIG. 57. It is to be noted that thelearning process of FIG. 57 is basically similar to the learning processof FIG. 52. In particular, a shift amount calculation process executedat step S173 of FIG. 57 is basically similar to the shift amountcalculation process of FIG. 12. It is to be noted, however, that thereliability determination process executed at step S22 of FIG. 12 andthe reliability calculation process (at step S32 of FIG. 18 and stepS112 of FIG. 44) in the motion compensating process selection processexecuted at step S23 are the reliability calculation process that doesnot use the input signal and the recreation signal described hereinabovewith reference to FIGS. 33, 37, 39 and 42. In other words, the shiftamount arithmetic operation section 313 of FIG. 56 does not execute thereliability calculation process (FIGS. 20 and 26), which uses the inputsignal and the creation signal.

Consequently, the shift amount detection characteristic amount outputtogether with the shift amount in the shift amount calculation processat step S173 of FIG. 57 does not include information such as a motioncompensation residual, which is produced based on the input signal andthe creation signal. Accordingly, at step S174, a class tap isconstructed based on the shift amount detection characteristic amount(for example, the reliability of the motion compensating vector) otherthan the information such as a motion compensation residual, which isproduced based on the input signal and the creation signal. Then at stepS175, a characteristic amount is extracted, and a class code is producedat step S176, whereafter a prediction tap is constructed at step S177.Then at step S178, the learning apparatus 301 uses the prediction tapand corresponding teacher data to learn the relationship between theteacher data and student data based on the class code from theclassification section 314 and predicts a prediction coefficient fromthe parameter B to arithmetically operate and produce predictioncoefficient data for each class. Then, the learning apparatus 301 storesthe prediction coefficient data into the coefficient memories 316-0 to316-N in accordance with the parameter B.

The prediction coefficient data for the individual classes stored in thecoefficient memories 316-0 to 316-N in such a manner as described aboveare stored into the coefficient memories 71-0 to 71-N of the predictioncoefficient selection section 53 of FIG. 53.

FIG. 58 shows a third example of configuration of the classificationadaptation processing section 31 of FIG. 1. It is to be noted that, inFIG. 58, like elements to those of FIG. 2 are denoted by like referencecharacters.

In the example of FIG. 58, the classification adaptation processingsection 31 that does not involve an intra-frame process (a process thatdoes not refer to any other picture, that is, a process that does notexecute motion compensation prediction) is shown. In the classificationadaptation processing section 31 of FIG. 58, the input signal is inputto the classification section 52 and the prediction arithmetic operationsection 54. The classification section 52 uses the input signal toconstruct a class tap, produces a class code based on an extractedcharacteristic amount and outputs the class code to the predictioncoefficient selection section 53.

The prediction arithmetic operation section 54 uses the input signal toconstruct a prediction tap, uses prediction coefficient data from theprediction coefficient selection section 53 to execute an arithmeticoperation process based on the prediction tap, performs an intra-frameprocess to create a time resolution and outputs a creation signal of thecreated time resolution. Accordingly, the classification adaptationprocessing section 31 of FIG. 58 does not use the creation signal.

Subsequently, the creation process executed by the classificationadaptation processing section 31 of FIG. 58 is described with referenceto a flow chart of FIG. 59.

The classification section 52 waits until the input signal is inputthereto at step S201. If the input signal is input, then theclassification section 52 uses the input signal to construct a class tapin accordance with the noticed pixel at step S202.

At step S203, the classification section 52 extracts a characteristicamount of a pixel, which forms the constructed class tap. Then at stepS204, the classification section 52 produces a class code by a 1-bitADRC process or the like based on the extracted characteristic amountand outputs the class code to the prediction coefficient selectionsection 53.

At step S205, the prediction coefficient selection section 53 selectsone of the coefficient memories 71-0 to 71-N corresponding to theparameter B output from the parameter control section 27. The predictioncoefficient selection section 53 selects, from among the predictioncoefficient data stored in the selected coefficient memory, predictiondata corresponding to the class code output from the classificationsection 52, and outputs the selected prediction coefficient data to theprediction arithmetic operation section 54. It is to be noted that suchprediction coefficient data are calculated by a learning apparatus 301,which is hereinafter described with reference to FIG. 60, and stored inadvance in the coefficient memories 71-0 to 71-N in accordance with theparameter B.

The prediction arithmetic operation section 54 constructs a predictiontap in accordance with the noticed pixel from the input signal at stepS206. Then at step S207, the prediction arithmetic operation section 54uses the prediction coefficient data selected by the predictioncoefficient selection section 53 to perform a prediction arithmeticoperation process to produce a creation signal and outputs the creationsignal.

FIG. 60 shows an example of configuration of the learning apparatus 301that learns prediction coefficient data for individual classes to bestored into the coefficient memories 71-0 to 71-N built in theprediction coefficient selection section 53 of the classificationadaptation processing section 31 of FIG. 58. In the example of FIG. 60,an example of configuration of the classification adaptation processingsection 31 where an intra-frame process is executed is shown. It is tobe noted that like elements to those in FIG. 51 are denoted by likereference characters.

The learning apparatus 301 of the example of FIG. 60 receives only theinput signal but does not receive the creation signal as an inputthereto.

The classification section 314 of FIG. 60 is configured basicallysimilarly to the classification section 52 of FIG. 58. Theclassification section 314 uses a student signal which is a decodedsignal supplied from the decoding section 312, to construct a class tapand outputs a class code to the prediction coefficient calculationsection 315.

The prediction coefficient calculation section 315 uses the studentsignal to construct a prediction tap, and uses the prediction tap andthe corresponding teacher signal (input signal) to learn therelationship between the teacher signal and the student signal based onthe class code from the classification section 314. Then, the predictioncoefficient calculation section 315 predicts a prediction coefficientfor each parameter B to arithmetically operate and produce predictioncoefficient data for the individual classes, and stores the predictioncoefficient data into the coefficient memories 316-0 to 316-N inaccordance with the parameter B.

Now, a learning process of the learning apparatus 301 of FIG. 60 isdescribed with reference to a flow chart of FIG. 61.

At step S231, the encoding section 311 encodes the input signal andoutputs the encoded signal to the decoding section 312. At step S232,the decoding section 312 decodes the encoded signal in accordance withthe parameter B and outputs the decoded signal as a student signal tothe classification section 314 and the prediction coefficientcalculation section 315.

The classification section 314 uses the student signal to construct aclass tap at step S233 and extracts a characteristic amount of the classtap based on the student signal at step S234. At step S235, theclassification section 314 produces a class code based on the extractedcharacteristic amount of the class tap and outputs the class code to theprediction coefficient calculation section 315.

At step S236, the prediction coefficient calculation section 315 usesthe student signal to construct a prediction tap. At step S237, theprediction coefficient calculation section 315 uses the prediction tapand the corresponding teacher signal to learn the relationship betweenthe teacher signal and the student signal based on the class code fromthe classification section 314 and predicts a prediction coefficientfrom the parameter B to arithmetically operate and produce predictioncoefficient data for each class. Then at step S238, the predictioncoefficient calculation section 315 stores such prediction coefficientdata into the coefficient memories 316-0 to 316-N in accordance with theparameter B and then ends its processing.

The prediction coefficient data for the individual classes stored in thecoefficient memories 316-0 to 316-N in accordance with the parameter Bin this manner are stored into the coefficient memories 71-0 to 71-N ofthe prediction coefficient selection section 53 of FIG. 60.

It is to be noted that the image data processing apparatus of theexemplary embodiment is applied to DVD recording and/or reproductionapparatus, BS digital signal reception apparatus and the like.

The series of processes described above may be executed by hardware orby software. In the latter case, the image data processing apparatus 1of FIG. 1 is formed from, for example, such an image data processingapparatus 401 as shown in FIG. 62.

Referring to FIG. 62, a CPU (Central Processing Unit) 411 executesvarious processes in accordance with a program stored in a ROM (ReadOnly Memory) 412 or a program loaded in a RAM (Random Access Memory) 413from a storage section 418. Also data necessary for the CPU 411 toexecute various processes and so forth are suitably stored into the RAM413.

The CPU 411, ROM 412 and RAM 413 are connected to each other by a bus414. Also an input/output interface 415 is connected to the bus 414.

Connected to the input/output interface 415 are an input section 416including a keyboard, a mouse and so forth, an output section 417including a display unit in the form of a CRT (Cathode Ray Tube), an LCD(Liquid Crystal Display) unit or the like and a speaker or the like, astorage section 418 including a hard disk or the like, and acommunication section 419 including a modem, a terminal adapter and soforth. The communication section 419 performs a communication processthrough a network not shown.

A drive 420 is connected the input/output interface 415 when necessary.A magnetic disk 421, an optical disk 422, a magneto-optical disk 423, asemiconductor memory 424 or some other storage medium is suitably loadedinto the drive 420, and a computer program read out from the thus loadedstorage medium by the drive 420 is installed into the communicationsection 419 when necessary.

Where the series of processes are executed by software, a program thatconstructs the software is installed from a network or a recordingmedium into a computer incorporated in hardware for exclusive use or,for example, a personal computer for universal use, which can executevarious functions by installing various programs.

The recording medium is formed as a package medium having the programrecorded thereon or therein and distributed to a user in order toprovide the program separately from a computer such as, as shown in FIG.62, a magnetic disk 421 (including a floppy disk), an optical disk 422(including a CD-ROM [Compact Disk-Read Only Memory] and a DVD [DigitalVersatile Disk]), or a magneto-optical disk 423 (including an MD[Mini-Disk] [trademark]), or a semiconductor memory 334. The recordingmedium is also provided to a user in a state wherein it is incorporatedin an apparatus body in advance by forming as a ROM in which the programis recorded or hard disk included in the communication section 419.

It is to be noted that, in the present specification, the steps thatdescribe the program recorded in a recording medium may be but need notnecessarily be processed in a time series in the order as described, andinclude processes that are executed in parallel or individually withoutbeing processed in a time series.

As described above, according to the exemplary embodiment, timeresolution can be performed. Particularly, time resolution of a highquality can be performed. Further according to the present invention,coding distortion such as block distortion or mosquito noise issuppressed.

While a preferred embodiment of the present invention has been describedusing specific terms, such description is for illustrative purpose only,and it is to be understood that changes and variations may be madewithout departing from the spirit or scope of the following claims.

Thus, the foregoing discussion discloses and describes merely exemplaryembodiment of the present invention. As will be understood by thoseskilled in the art, the present invention may be embodied in otherspecific forms without departing from the spirit or essentialcharacteristics thereof. Accordingly, the disclosure of the presentinvention is intended to be illustrative, but not limiting of the scopeof the invention, as well as the claims. The disclosure, including anyreadily discernible variants of the teachings herein, define, in part,the scope of the foregoing claim terminology such that no inventivesubject matter is dedicated to the public.

This Application claims the benefit of priority document JP 2002-371403,filed in Japan on Dec. 24, 2002, the entire contents of which areincorporated by reference herein in its entirety.

1. An image data processing apparatus, comprising: selection means for selecting a motion compensating vector of a noticed region based on additional information added to student data; classification means for classifying the noticed region into one of a plurality of classes based on the motion compensating vector selected by said selection means; and learning means for constructing a prediction tap of the noticed region based on the motion compensating vector selected by said selection means and learning a prediction coefficient based on the class classified by said classification means using teacher data corresponding to the constructed learning tap.
 2. An image data processing apparatus according to claim 1, further comprising: tap extraction means for tap extracting the noticed region as a tap based on the motion compensating vector selected by said selection means, and said classification means classifies the noticed region into one of the plurality of classes based on a positional relationship of the noticed region drawn near as a tap by said tap extraction means and a boundary of the noticed region before drawn near as a tap by said tap extraction means.
 3. An image data processing method, comprising: selecting a motion compensating vector of a noticed region based on additional information added to student data; classifying the noticed region into one of a plurality of classes based on the motion compensating vector selected by the process of the selection step; and constructing a prediction tap of the noticed region based on the motion compensating vector selected by the process of the selection step and learning a prediction coefficient based on the class classified by the process of the classification step using teacher data corresponding to the constructed learning tap.
 4. A computer readable carrier including computer program instructions that cause a computer to implement a method of creating a time resolution for an image data signal on which a computer-readable program is recorded, comprising: selecting a motion compensating vector of a noticed region based on additional information added to student data; classifying the noticed region into one of a plurality of classes based on the motion compensating vector selected by the process of the selection step; and constructing a prediction tap of the noticed region based on the motion compensating vector selected by the process of the selection step and learning a prediction coefficient based on the class classified by the process of the classification step using teacher data corresponding to the constructed learning tap.
 5. An instruction set that causes a computer to execute a time resolution of an image data signal, comprising: selecting a motion compensating vector of a noticed region based on additional information added to student data; classifying the noticed region into one of a plurality of classes based on the motion compensating vector selected by the process of the selection step; and constructing a prediction tap of the noticed region based on the motion compensating vector selected by the process of the selection step and learning a prediction coefficient based on the class classified by the process of the classification step using teacher data corresponding to the constructed learning tap.
 6. An image data processing apparatus, comprising: a selection unit, selecting a motion compensating vector of a noticed region based on additional information added to student data; a classification unit, classifying the noticed region into one of a plurality of classes based on the motion compensating vector selected by said selection unit; and a learning unit, constructing a prediction tap of the noticed region based on the motion compensating vector selected by said selection unit and learning a prediction coefficient based on the class classified by said classification unit using teacher data corresponding to the constructed learning tap. 